Systems and methods for processing 2d/3d data for structures of interest in a scene and wireframes generated therefrom

ABSTRACT

The inventions herein relate generally to improvements in the generation of wireframe renderings derived from 2D and/or 3D data that includes at least one structure of interest in a scene. Such wireframe renderings and similar formats can be used in, among other things, 2D/3D CAD drawings, designs, drafts, models, building information models, augmented reality or virtual reality, and the like. Measurements, dimensions, geometric information, and semantic information generated according to the inventive methods can be accurate in relation to the actual structures. The wireframe renderings can be generated from a combination of a plurality of 2D images and point clouds, processing of point clouds to generate virtual/synthetic views to be used with the point clouds, or from 2D image data that has been processed in a machine learning process to generate 3D data. In some aspects, the wireframe renderings are accurate in relation to the actual structure of interest, automatically generated, or both.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation application claiming priority to, andthe benefit of, U.S. Non-Provisional application Ser. No. 15/881,795filed Jan. 28, 2018, which claims priority to U.S. ProvisionalApplication No. 62/451,700 filed Jan. 28, 2017. The contents of bothapplications are incorporated herein in their entireties by thisreference.

STATEMENT OF GOVERNMENT INTEREST

This invention was made with government support under contract numbers1519971 and 1632248 awarded by the National Science Foundation. TheGovernment has certain rights to the invention.

FIELD OF THE INVENTION

The inventions herein relate generally to improvements in the generationof wireframe renderings derived from 2D and/or 3D data that includes atleast one structure of interest in a scene. Such wireframe renderingsand similar formats can be used in, among other things, 2D/3D CADdrawings, designs, drafts, models, building information models,augmented reality or virtual reality, and the like. Measurements,dimensions, geometric information, and semantic information generatedaccording to the inventive methods can be accurate in relation to theactual structures. The wireframe renderings can be generated from acombination of a plurality of 2D images and point clouds, processing ofpoint clouds to generate virtual/synthetic views to be used with thepoint clouds, or from 2D image data that has been processed in a machinelearning process to generate 3D data. In some aspects, the wireframerenderings are accurate in relation to the actual structure of interest,automatically generated, or both.

BACKGROUND OF THE INVENTION

Digital three dimensional (“3D”) building models, such as wireframerenderings, are increasingly used for diverse tasks in architecture anddesign such as construction planning, visualization, navigation,simulation, facility management, renovation, and retrofitting. To thisend, 3D data relating to objects, structures, or elements in a scene,such as point clouds, have utility in a variety of applications. As oneexample, when manipulated to generate a wireframe rendering, 3D data inthe form of point clouds can be used in CAD systems, buildinginformation model (“BIM”) systems, A/R (“augmented reality”) or V/R(“virtual reality”) or the like. For example, point clouds generatedfrom LiDAR techniques have typically been used as a starting point forgenerating 3D models for use in various engineering-type applications.However, current methodologies do not readily allow 3D data derived fromstructures, objects, or elements in a scene to be automaticallyprocessed to generate wireframe renderings that are accurate in relationto the structure or object for which the renderings are being generated.In this regard, while computer vision technology has been advancingquickly in recent years, the effectiveness and quality of algorithms andtechniques to extract information from imaging of structures (or objectsetc.), such as buildings, has not progressed to ensure high qualityresults can be obtained using automatic methods, especially when thestructure is even modestly complex in form. Existing algorithms used toanalyze and extract wireframe renderings in suitable detail from 3Ddata, such as point clouds, are often unable to fully resolve thegeometric features for structures, objects, or elements present in ascene without also an attendant manual evaluation and/or manualmanipulation of the output. This means that existing algorithms areincapable of automatically generating high quality reconstructions formany structures and/or objects that are of interest.

Lack of accurate wireframe renderings from 3D data using automaticmethods is particularly acute when the structure being modeled from 3Ddata is from a time that building designs were generated usingnon-computer methods because there is likely no design or engineeringinformation retrievable for use as 3D data. This is a category thatincludes the vast majority of buildings in the world. To generate highquality wireframe renderings of such structures such that they can beused for engineering-related applications, such as repair andrehabilitation, time consuming manual activities must be conducted.

Another problem faced in generation of accurate wireframe renderingsfrom 3D data is when a structure or object is “arbitrary,” in that it isnon-standard and/or has not previously been analyzed in the samecontext. Such arbitrary structures or objects will therefore not haveaccurate baseline information retrievable in an available library ofstructural elements or other relevant information to provide referencefor automatic generation of accurate wireframe renderings. To obtainhigh quality wireframe renderings for these arbitrary structures orobjects, human supervision via manual interventions is required. Costand complexity of creating high quality wireframe renderings for objectsthat are being examined for the first time is therefore high today, dueto the requirement that manual interventions be conducted in order togenerate high quality wireframe renderings using current state of theart methodology.

An overview of prior art methodologies and the shortcomings in thespecific context of building model generation are highlighted in therecent article “A Hybrid Approach for Three-Dimensional BuildingReconstruction in Indianapolis from LiDAR Data,” Remote Sens., 9, Mar.2017, 310. This publication, which is incorporated herein in itsentirety by this reference, outlines three existing approaches tomodeling of buildings using 3D data generated from LiDAR, the contoursof which can generally be applied to modeling of objects other thanbuildings and 3D data derived from sources other than LiDAR. Tosummarize this discussion, building modeling generally includes twosteps: 1) the detection of building boundaries from 3D data; and 2) thereconstruction of building models from such data. For “data-driven”(also called “bottom-up”) methodologies, buildings are considered to bean aggregation of roof planes represented in the point clouds followedby processing via segmentation using one or more of a variety ofalgorithms such as, region growing, RANSAC, clustering, or ridge or edgebased. For model-driven (also called “top-down”) approaches, parametricbuilding models are generated and stored to generate 3D models orwireframes from the point clouds. The data-driven approach has theadvantage of detecting basic elements of the building, but the qualityof 3D modeling or wireframe generation can be limited by the algorithmapplied for segmentation. Automatic 3D modeling or wireframe generationis nonetheless problematic when small features are present in thestructure and/or substructures are contained within larger structures.The model-driven approach has the advantage that, for roof 3D modelingor wireframe generation, the various facets are standardized, whichimproves 3D modeling or wireframe generation quality. However, themodel-driven approach requires that the needed model element is storedin the library, which makes not standard roof or non-ideal 3D modelingor wireframe generation problematic.

A “hybrid approach” is a combination of the data-driven and model-drivenapproaches. This methodology first generates the features of thestructure of interest using algorithms as discussed previously, followedby application of previously generated models to provide the 3D buildinginformation, for example, in the form of a wireframe rendering. Whilethe hybrid approach can produce greater wireframe quality than use ofeither the data-driven or model-driven approaches individually, theinherent problems in each of the approaches (e.g., algorithm quality andlibrary limitations) nonetheless reduce the overall 3D modeling andwireframe rendering quality of these methods. While the hybrid approachhas been applied using automatic methods with some success for simplestructures where simple building structures are applicable and for whichpre-defined building primitives can be applied, for structures andobjects that where non-planar and/or a plurality of planar buildingfeatures are present, wireframe renderings that do not require at leastsome manual interaction and/or manual result validation remain elusive.

As shown, significant progress remains to be made in the generation ofwireframe renderings for buildings, and other structure types, as wellas for the generation of wireframe renderings for other object types. Inthis regard, while some percentage of buildings are likely to compriseat least some standard features (e.g., roof parts, windows, doors,etc.), many buildings comprise non-standard features or complexarrangements of standard features (e.g., multi-faceted roof structures)that will be difficult to accurately resolve due to the complexity ofthe arrangements therein. Further, non-building structures in a sceneare more likely to comprise non-standard components or otherwise bearbitrary. Accordingly, current state of the art methodologies cannotsuitably generate accurate wireframe renderings for such structures andobjects without manual intervention in the generation of the 3Dinformation and/or manual validation of the results thereof.

Moreover, even for wireframe renderings that are generated with at leastsome human supervision, that is, are not generated fully automatically,there remains a need for accurate measurements, dimensions, geometricinformation, topological information, semantic information, and thelike, where such accurate measurements etc. are close in value, or evensubstantially equivalent to, those for the actual structures or objects.It follows that it would be even better to be able to generate suchaccurate measurements etc. automatically.

There remains a need for improvements in methods to generate wireframerenderings from 3D data associated with structures of interest in ascene, where such structures comprise buildings, parts of buildings, orother objects. The present invention provides these and other benefits.

SUMMARY OF THE INVENTION

The present invention relates to systems and methods for generatingwireframe renderings of one or more structures of interest in a scene.The structure of interest optionally comprises at least one non-planarsurface. The wireframe renderings can be generated automatically. Thewireframe renderings can be accurate in relation to the actualstructure(s) of interest in the scene. Still further, the wireframerenderings can be automatically generated and accurate in relation tothe actual structure(s) of interest in the scene.

In accordance with implementations of the present inventions, 2D and 3Ddata is provided for at least one structure of interest. The 2D and 3Ddata can comprise, for example, a plurality of 2D images including theat least one structure of interest and at least one point cloudincluding the at least one structure of interest, wherein the pluralityof 2D images are associated with the at least one point cloud. Yetfurther, the 2D and 3D data can comprise, for example, 2D data generatedfrom the processing a plurality of point clouds that incorporate the atleast one structure of interest. Such 2D and 3D data is processed togenerate 3D data comprising boundary information associated with thestructure. From such processed information, at least one geometricprimitive is extracted therefrom, where the extracted geometricprimitive(s) are selected from a specific list as set out in Table 1hereinafter. From the at least one extracted geometric primitives, awireframe rendering of at least part of the structure of interest in thescene is provided. Yet further, at least two geometric primitives can beextracted from the processed information, wherein all or part of thefirst geometric primitive is within the second geometric primitiveboundary. Still further, a plurality of geometric primitives can beextracted from the processed 3D information, and the plurality ofgeometric primitives can be combined to generate the wireframerendering.

The generated wireframe renderings can be input into a machine learningtraining set associated with a library of information relevant to thestructure of interest. Information associated with the library ofinformation can be used in subsequent wireframe generation processes.

The types of structures of interest that can be used with the inventionsherein are expansive. For example, the structures can comprise buildingexteriors or interiors. The structure of interest can comprise a roof.In this regard, the wireframe rendering can be used, for example, in aroofing report. Yet further, the generated wireframes can be used in,among other things, 2D/3D CAD drawings, designs, drafts, models,building information models, augmented reality or virtual reality, andthe like.

In a further aspect, wireframe renderings of the at least one structureof interest can be generated directly from 2D data incorporating thestructure of interest in a scene without first generating a point cloudas an interim step. In this aspect, machine learning models and computervision techniques are utilized to generate 3D data from which wireframerenderings can be generated using substantially only images and/orcamera parameters derived from the 2D images.

Additional advantages of the invention will be set forth in part in thedescription that follows, and in part will be apparent from thedescription, or may be learned by practice of the invention. Theadvantages of the invention will be realized and attained by means ofthe elements and combination particularly pointed out in the appendedclaims. It is to be understood that both the foregoing generaldescription and the following detailed description are exemplary andexplanatory only and are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 illustrates an exemplary process for generating wireframerenderings from 2D and 3D data incorporating a structure of interest.

FIG. 2 illustrates an exemplary process for generating wireframerenderings directly from 2D data.

DETAILED DESCRIPTION OF THE INVENTION

Many aspects of the disclosure can be better understood with referenceto the Figures presented herewith. The Figures are intended toillustrate the various features of the present disclosure. While severalimplementations may be described in connection with the includeddrawings, there is no intent to limit the disclosure to theimplementations disclosed herein. To the contrary, the intent is tocover all alternatives, modifications, and equivalents.

The term “substantially” is meant to permit deviations from thedescriptive term that do not negatively impact the intended purpose. Alldescriptive terms used herein are implicitly understood to be modifiedby the word “substantially,” even if the descriptive term is notexplicitly modified by the word “substantially.”

The term “about” is meant to account for variations due to experimentalerror. All measurements or numbers are implicitly understood to bemodified by the word about, even if the measurement or number is notexplicitly modified by the word about.

“2D data,” in one aspect, are derived from image data as describedhereinafter. Briefly, 2D image data can be derived from image capturedevices, for example. In another aspect, 2D data is generated in theform of virtual views derived from point clouds are discussedhereinafter.

“3D data” means raw data such as 3D point clouds, or mesh or vectorfiles. 3D data comprising information regarding structures or objects ofinterest in a scene can be directly generated from active scanningtechnologies such as LiDAR, laser scanners, time-of-flight cameras, PMD(Photonic Mixer Device) cameras, depth-sensing cameras, and the like. 3Ddata in the form of point clouds can also be obtained from stereovisionphotogrammetry using two passive image capturing devices. More recently,passive single vision photogrammetry, which can be obtained from regularor video imaging techniques, has been disclosed to create point clouds,as described in U.S. Pat. No. 9,460,517 (the “'517 patent”), thedisclosure of which is incorporated herein in its entirety by thisreference. More detailed discussion of the manners in which the 3D datacan be generated for use herein is set out herein below.

3D data useable in the processes herein can also be derived from 2D datain an inventive methodology where a plurality of 2D images incorporatingthe structure or object of interest are processed via the application ofcomputer vision techniques, machine learning techniques, or the like. Inthis way, information about geometry, objects of interest, interactionof objects in a scene, and their semantic information can be determinedin 3D space using only 2D data. As a non-limiting example, multiple 2Dimages including the same object of interest in the scene from two ormore different viewpoints or perspectives, such as a window, can betriangulated into 3D space and represented with size and distancerelationships to one or more other objects present in the scene. Usingsuch processes that can operate to impart special and dimensionalcharacteristics to structures or objects present in 2D images, 3D datacan be created for use in the inventive processes herein. Accordingly, aform of 3D data for use in the invention can be synthesized from 2Dimage data, in a significant context. More detail about such methodologyis provided hereinafter.

As used herein, a “wireframe rendering” refers to one or more of avisual, abstract, data-interchange, human-readable, machine-readable,text-based, and/or object-based 3D representation of a physical object.This 3D representation can be created by specifying each edge of thephysical object where two mathematically continuous smooth surfacesmeet, or by connecting an object's constituent vertices using straightlines or curves. In other words, the wireframe rendering can be an edgeor skeletal representation of a real-world object which consists ofpoints, lines, arcs, circle, and other curves that define the edges orcenter lines of objects. Still further a wireframe rendering can also bedefined as any combination of representations of a structure, objectetc. of a collection of surface boundaries, points of interest in 2D or3D, line segments in 2D or 3D, and/or lines connecting points ofinterest in 2D or 3D, generated from a structure of interest in a scene.A wireframe rendering can be part of or useable in the context of one ormore of 2D/3D CAD drawings, designs, drafts, models, A/R, V/R of one ormore structures of interest in a scene. In an illustrative example,according to these definitions, a 3D wireframe rendering could be usedto generate basic 3D designs for evaluation and fast design iterations,to view the model from any viewpoint, to analyze spatial relationshipsincluding the distances between corners and edges as well as checkingvisually for potential interferences, to generate perspective views, togenerate standard orthographic and auxiliary views automatically, and toact as reference geometry for 3D solid, surface, mesh modeling. Infurther examples, a wireframe rendering can include an object-basedrepresentation of a real-world scene in which each object in the scenerepresents a data structure that comprising information about all orpart of the geometry, topology, and semantic attributes of an object.This object-based representation is typically referred to as “BIM” andcould have multiple levels of development (LOD) depending on the levelof details that are stored in the object data structure.

The term “wireframe rendering” may be used herein to have the samemeaning as “wireframe,” in some contexts.

As would be recognized, a surface boundary is a “concave hull” thatencapsulates the collection of 3D data points belonging to a surface.Concave hulls can also be termed as “alpha shapes.” A generatedwireframe rendering merges and shows overlapping portions or segments ofsurface boundaries that are derived from a collection of surfaces,wherein such overlapping portions are provided in the generatedwireframe rendering as a single curve that is shared among all of theintersecting surfaces.

A “structure” as used herein, when presented in the context of theinvention herein, can be represented or conceptualized as a collectionof features in the form of surfaces having boundaries. In accordancewith the inventive methodology herein, in some aspects, each of thesurfaces in a collection of surfaces can, independently, be extractableas one of several forms of mathematically defined “geometricprimitives,” as such several forms are described in more detailhereinafter in relation to Table 1. These geometric primitives haveboundary information from which wireframe renderings can be derived.

As used herein, a “scene” is a location that can incorporate one or morestructures of interest.

A “structure of interest” can encompass a wide variety of structures(that also may be referred to in some contexts as “objects” or“elements”) that may be present in a scene such as, for example,components of a building (e.g., doors, windows, walls, roofing, façade,stairs, plumbing/piping, electrical equipment, floors, flooringmaterial, decorative aspects), landscape components, and the like.Indeed, a “structure of interest” can be anything from which 3D datasuitable for processing into wireframe renderings can be derived.Further, the methodology herein can be utilized to extract informationabout more than one structure of interest in a scene, such as twodifferent structures (e.g., a door and a window); or a collection ofsmaller structures—or “sub-structures/elements” (e.g., doors, windows,etc.)—associated with a larger structures (e.g., all or part of abuilding) where information about such collection of smaller and largerstructures can be processed into one or more wireframe renderingsaccording to the inventive methodology herein.

The definition of “structure” is meant to be expansive to include anyelement or feature present in a scene that might be relevant to a user.For example, a scene can include one or more structures of interest,where the scene comprises an exterior location. The structures in thescene can comprise, for example, doors, windows, roofs, facades,landscape elements, etc. Each of these structures can be comprised ofsub-structures/elements, such as panes in a window, or a door knob or akick plate on a door. In a further example, a pitched roof on a housecan comprise a structure of interest, and/or the various sub-parts ofthe roof can also comprise a plurality of structures of interest orsub-structures/elements of interest in conjunction with the entire roofstructure (e.g., skylights, “eyebrows” etc.). In some aspects, wireframerenderings generated from the scene can incorporate only part of theoverall structure, such as part of a roof. Such partial structureinformation as provided in the wireframe rendering may nonetheless besuitable for use in one or more applications.

In another non-limiting example, a scene can comprise one or morestructures of interest located in the interior of a building orfacility, where the one or more structures can comprise walls, floors,mechanical equipment, windows, doors, doorways, furniture (e.g., chairs,tables, etc.), fixtures (e.g., lighting, cabinets, flooring), computers,electronic equipment, inventoriable items regardless of size, value etc.Each of these structures can comprise sub-structures/elements. Forexample, when the structure of interest is an HVAC system,sub-structures of interest may include the mechanical components (e.g.,compressor, air exchanger etc.), duct work, and the like, all or some ofwhich can be incorporated in one or more wireframes, as set out in moredetail herein.

In some aspects, each of the surfaces or each of the surface boundariesfor the structure of interest derived from the 2D and/or 3D data can beincorporated into one or more wireframe renderings. In other aspects,only some of the surfaces or some of the surface boundaries for thestructure of interest can be incorporated into one or more wireframerenderings. In some aspects, only one of the surfaces or surfaceboundaries of one or more structures may be of interest. As such, thegenerated wireframe rendering may comprise information about all or onlypart of a structure of interest.

“A least part of” in reference to the one or more structures of interestrefers to an instance where only a portion or portions but not theentire structure is incorporated in the generated wireframe. In someaspects, the present invention can be utilized to generate informationabout structures of interest that may be present or partially present ina scene, as desired by a user. An example of when “at least part of” ispresent in a scene can include instances where the structure, andtherefore any surfaces and surface boundaries associated therewith, isonly partially captured for generation into a wireframe, due toocclusion or the perspective of the image capture device or where partsof the structure fit into the scene but the entirety of the structuredoes not.

In broad aspects, the present invention provides methods to generatewireframe renderings for one or more structures of interest in a scenefrom 3D data and/or 2D data that includes the structure of interest,where the structure optionally comprises at least one non-planar elementtherein. The 3D and/or 2D data are processed to generate 3D datacomprising boundary information for the structure, where this generated3D data having boundary information is associated with the structure ofinterest. Next, at least one geometric primitive is extracted from thegenerated 3D data having boundary information, where each of thegeometric primitives extracted therefrom are each, independently,selected from the list of Table 1.

TABLE 1 No. Type Canonical Expression 1 One real plane ax + by + cz + d= 0 2 Ellipsoid x²/a + y²/b + z²/c = 1 3 Elliptic cylinder x²/a + y²/b =1 4 Hyperbolic cylinder x²/a − y²/b = 1 5 Parabolic cylinder x² + 2y = 06 Quadric cone x²/a + y²/b − z²/c = 0 7 Hyperboloid of one x²/a + y²/b −z²/c = 1 sheet 8 Hyperboloid of two x²/a + y²/b − z²/c = −1 sheets 9Hyperbolic paraboloid x²/a − y²/b + 2z = 0 10 Elliptic paraboloid x²/a −y²/b + 2z = 0 11 Intersecting planes x²/a − y²/b = 0 12 Parallel planesx² = 1

3D data used in the extraction of the listed geometric primitivesincorporates information about the structure(s) of interest, such as, innon-limiting examples, vertices, line segments, connections betweenobjects of interest, edges, surfaces and surface boundaries of all orpart of one or more structures of interest.

In some aspects, the wireframe renderings are generated about thestructure of interest automatically, that is, without human interventionor manual manipulation by a user. In other aspects, the wireframerenderings are generated about the structures of interest substantiallyautomatically. In further aspects, the wireframe renderings aregenerated in a plurality of steps, wherein at least one of the steps isconducted without human supervision or the need for a human tomanipulate or validate the output of those steps or a collection ofsteps. Still further, the wireframe renderings are generated in aplurality of steps, wherein all of the steps are conducted without humansupervision or the need for a human to manipulate or validate the outputof those steps or a collection of steps.

Wireframe renderings of the one or more structures of interest canincorporate a selection step whereby the structure(s) of interest orparts of the structures of interest are selected (or identified) fromthe scene or from a collection of other structures that may be presentin the scene. Such selection/identification can be performedautomatically by a computer, by a user, or both by a computer and auser. Alternatively, the structure that is not of interest or part ofthe structure that is not of interest can be deselected. Such selectionor deselection can be done automatically by the computer, manually by auser, or both automatically or manually.

In some aspects, the present invention allows one or more structures ofinterest or part of one or more structures present in a scene to berendered as wireframes substantially in a single, automatic, operation.Such wireframe renderings are accurate models of the structure ofinterest as well as, in some aspect, numerically accurate. “Modelaccuracy” means that the wireframe rendering comprises the samestructural element and is recognizable as the object type as is theactual structure of interest in the scene. “Numerical accuracy” meansthat any numerical values derivable from the 3D information are close invalue to the actual numerical values for the object of interest in thescene. Yet further, “accuracy” in relation to the invention herein meansthat one or more of the measurement, dimension, geometric information,topological information, semantic information, and labeling (that is,identification) is substantially equivalent to the actual (that is, reallife) measurements, dimensions, geometry, topology, semantic informationand labeling information for the actual object in the scene. In thisregard, the structure of interest in a scene will have anactual/real-life attribute, and the information generated from themethodology herein will be substantially the same as theactual/real-life attribute. When the information returned is numericalin nature (e.g., measurement, dimension, geometry information) thevalue(s) returned from the methodology herein will be within about 5% orabout 1% or about 0.1% or closer to the actual value. (Such numericalaccuracy is discussed in more detail hereinafter.) When the informationreturned is a label, topological relationship, or the like, the returnedinformation will be the same (e.g., a returned label of a “chair” willcorrectly identify a chair as being present in the scene).

As noted previously, the prior art comprises data-driven, model-driven,and hybrid approaches to the generation of wireframe renderings from 3Ddata generated from structures such as roofs, buildings, etc. Currently,model-driven or hybrid approaches are used for wireframe generationbecause data-driven approaches used alone are not able to generateaccurate results. In the non-limiting case of a roof, prior artmodel-driven methodologies cannot consistently generate accuratewireframe renderings of the roof or structure without substantial manualinteraction. This is due at least to the fact that, when generatingwireframes regarding structures of interest that comprise one or aplurality of surfaces, the frameworks of analysis primarily focus on amodel-driven approach that applies a library of previously defined andretrievable information identified as being pertinent to the structurebeing reconstructed, namely, a library of roof primitives, solidbuilding primitives, pre-defined roof shapes, or description of roofsusing a formal grammar. Put simply, the prior art assumes that thenecessary shapes—that is, the “high-level geometric primitives”referenced in the prior art—are available to be defined in the datagenerated from the structures to generate an accurate wireframerendering of that roof. For complex roofs and non-standard roofs andbuildings—or more broadly structures of interest in a scene havingcomplexity—accurate wireframe renderings cannot be generated therefromunless some form of human supervision is applied during the generationor for validation, without such human supervision, errors will commonlyresult because the necessary shapes to successfully generate each of thepieces to generate an accurate model are not available in thepre-defined library. Accordingly, with prior art model-driven methods,the resulting wireframe renderings typically must be reviewed by a humanone or more times during the generation thereof. If necessary, thegenerated wireframe rendering must be corrected or adjusted to allow thewireframes rendered therefrom to comprise information suitable for use.

In significant aspects, the present invention allows geometrically,dimensionally, topologically, and semantically accurate wireframes to begenerated directly from 2D and/or 3D data incorporating one or morestructures of interest in a scene. In further aspects, the inventivewireframe renderings can be generated substantially without the need fora manual wireframe validation and/or correction step. In some aspects, avalidation step can optionally be conducted, where such validation stepcan be done automatically, such as by use of information derived frommachine learning processes as discussed hereinafter.

In some cases, 3D data, such as point clouds, and 2D image dataincorporate the structure of interest can further incorporate astructure that is not of interest. For example, 3D data may include abuilding as a structure of interest and a tree that partially occludesthe building. 2D image data associated therewith, such as image datafrom which the point clouds are derived, will also comprise suchocclusions. In substantial aspects, the generated wireframe renderingwill comprise substantially only aspects, components, or featuresassociated with each of one or more structures of interest in the scene.In other aspects, the wireframe renderings generated herein willsubstantially exclude information about structure(s) present in thescene that are not of interest, such as the tree in this example.

In various aspects, the inventive methodology herein is able to extract,resolve, isolate, or segment the respective surfaces for the structuresof interest from other surfaces in the scene that are not part of thestructure of interest. In this regard, the present inventive methodologycan, for example, allow an accurate wireframe rendering to be extractedfor all or part of each of the structures of interest even when at leastpart of the 2D and/or 3D data incorporates data about a structure orobject not of interest that at least partially occludes the structure ofinterest. As an implementation of this inventive methodology, asubstantially accurate wireframe rendering can be provided of a roof,where such wireframe consists substantially only of boundary informationassociatable with the roof aspects present in the actual roof in thescene. Yet further, the generated wireframe will substantially excludesurfaces, lines, and vertices that are not part of the structure ofinterest. In some aspect, such exclusion can be done automatically andwithout the need for manual operation by a human operator.

The wireframe renderings generated according to the inventivemethodology are substantially accurate because they allow measurementsor other dimensional information to be derived from the generatedwireframes that have less than about 5% or less than about 2% or lessthan about 1% or less than about 0.1% or less than about 0.01% deviationfrom the measurements, dimensions, and geometry of the actual (e.g.,“real life”) structure of interest. In this regard, at least accuratelength, width, surface area, and volume measurements can be obtained.For example, if a roof section on a house has a length of 50 feet, thegenerated wireframe section corresponding to that roof length will begenerated as ranging in length of about 45 to about 55 feet, or fromabout 47.5 to about 52.5 or from about 49 to about 51 feet or from about49.5 to about 50.5 feet or even less for very accurate measurements.Similarly, the dimensions of each of the various roof sections can becalculated to thereby provide an accurate surface area of the roof, suchas within about 5% or about 2% or about 1% or about 0.1% or even or lessthan the actual surface area of the roof.

In some aspects, the wireframe renderings generated according to theinventive methodology are also substantially accurate because they allowfor counting and grouping of like items together where such countingand/or grouping are correct in relation to the real-life items. Forexample, if a structure has multiple roof facets or material types,labels returned for those aspects will conform to each of the rooffacets or each of the material types, as well as generating themeasurement information as discussed previously. For the case ofmultiple windows, the information generated from the methodology hereincan provide the location of each individual window, the count of totalwindows, the surface area of each window, the total surface area of allwindows, and/or the sections of the structures which contain a window.Such labeling information can be included on a wireframe rendering, andfurther may be provided for use in other output forms, such as roofingreports, etc. Roofing reports are described, for example, in one or moreof U.S. Pat. Nos. 8,078,436; 8,145,578; 8,170,840; 8,209,152; 8,515,125;8,825,454; 9,135,737; 8,670,961; 9,514,568; 8,818,770; 8,542,880;9,244,589; 9,329,749; 8,938,090 and 9,183,538, the disclosures of whichare incorporated herein in their entireties by this reference.

In significant aspects, the one or more structures of interest in thescene each, independently, include surfaces having surface boundariesthat can be represented as one or a plurality of primitive geometries asset out in previous Table I. One or a variety of the listed geometricprimitives can typically be extracted (or resolved or isolated) from theprocessed 3D and/or 2D data derived from the structure of interest togenerate wireframe renderings according to the methodology herein. Thatis, 3D and/or 2D data derived from each of the one or more structures ofinterest can be processed to generate 3D data having at least oneextractable geometric primitive of Table I from which a wireframerendering can be generated. When the structure of interest comprises aplurality of surfaces from which a plurality of the geometric primitivesof Table I can be extracted from processed 3D data having boundaryinformation, such geometric primitives can be the same or different. Yetfurther, a plurality of extracted geometric primitives of Table I can becombined to generate a wireframe rendering for a multi-dimensionalstructure of interest.

By way of explanation, the inventive methodology is a significantdeparture from existing paradigms used for wireframe rendering. Insteadof fitting externally generated geometric building primitives (i.e.,high-level geometric primitives) or other pre-defined structural libraryinformation to processed 3D data (e.g., point clouds) having a structureof interest in a scene incorporated therein, the inventors herein havediscovered that it is possible to extract one or more specific low-levelgeometric primitives from a finite list as set out in Table 1 in fromthe 3D data having boundary information therein and to extract suchprimitives to generate wireframe renderings that are accurate inrelation to the real-life structure in the scene. Still further, theinventors herein have determined that the finite set of low-levelgeometric primitives can present all of the necessary mathematicaloperations (including quadric surface equations) to generate a model ofany structure (e.g., building, roof, parts of a building, or any otherobject of interest) that is included in the 2D and/or 3D data that isgenerated and processed according to the methodology herein.

To illustrate the concept of geometric primitive extracted fromprocessed 3D data having boundary information in the present invention,a pitched roof will generally incorporate a plurality of planarsurfaces. For scenes in which planar surfaces are present—here aroof—wireframe renderings of the planar sections can be obtaineddirectly from the processed 3D data having boundary information therein.

With regard to the planar sections of a roof, a wireframe rendering canbe created by extracting one or a collection of geometric primitivesfrom the 3D data having boundary information therein, where such 3D datais provided in the form of a point cloud, etc. that incorporates 2D data(either derived from a plurality of 2D images or provided in the form of“virtual/synthetic view” data). From this 3D and 2D data, 3D data thatcomprises boundary information allows one or more of the listedgeometric primitives to be extracted (or isolated or resolved orsegmented) to allow wireframe renderings to be generated.

In a further example, a wireframe rendering can be generated byextracting information from only 2D images, without the necessity tofirst generate a point cloud, wherein the extracted 2D informationcorresponds to the respective boundaries of each edge of the planarsections derivable from the scene. In this implementation, the 2D imagedata is directed into a machine learning process, as discussed furtherhereinafter.

In further aspects, fewer geometric primitives than listed Table I,specifically a subset of those listed, can be used to generate thewireframe renderings. In this regard, in some aspects, the geometricprimitives extractable from the processed 3D data comprise the list ofgeometric primitives of Table 2.

TABLE 2 No. Type Canonical Expression 1 One real plane ax + by + cz + d= 0 2 Ellipsoid x²/a + y²/b + z²/c = 1 4 Hyperbolic cylinder x²/a − y²/b= 1 6 Quadric cone x²/a + y²/b − z²/c = 0

In some aspects, the structure of interest in the scene comprises atleast one non-planar element therein, thus providing at least onenon-planar surface extractable (or isolatable, or resolvable, orsegmentable) therefrom. In further aspects, the structure of interest ina scene optionally comprises at least one non-planar element, thusproviding at least one non-planar surface extractable (or isolatable, orresolvable, or segmentable) therefrom. Yet further, the structure ofinterest comprises at least one substructure that is incorporated withinanother structure, such as a dormer located within a larger roofstructure. Such sub-structure can or can optionally comprise at leastone non-planar element, thus providing at least one non-planar surfaceextractable (or isolatable or resolvable or segmentable) therefrom.

As noted, in some aspects, the structure of interest can comprise atleast one non-planar element, thus providing at least one non-planarsurface identifiable, isolatable, resolvable, or segmentable therefrom.Accordingly, at least one non-planar geometric primitive, such as onehaving a quadric functionality, will be extractable (or isolatable orresolvable or segmentable) from the processed 3D data having boundaryinformation. For geometric primitives other than planar forms, that is,the non-planar geometric primitives set out above in Table 1 and Table2, the generated wireframe rendering will conform to the boundaries ofthat specific primitive(s) isolatable from the processed 3D data. Forexample, a geometric primitive of an ellipsoid surface can be extracted(or isolated or resolved or segmented) from the 3D data having boundaryinformation therein in the case of a roof structure that includes a domeshape therein. Still further, a combination of planar and ellipsoidsurfaces can be extractable (or isolatable or resolvable or segmentable)as geometric primitives from the processed 3D data to generate awireframe rendering of a roof structure that includes a plurality ofintersecting roof planes and a 3D dome.

As another example of the inventions herein, a concrete girder boxbridge could comprise the structure of interest in a scene. Such abridge could comprise, for example, one or more cylindrical columns, adeck having intersecting planar surfaces, one or more arch surfaces forconnecting the columns to the deck, etc. Using the methodology herein, aplurality of the specified geometric primitives from Table 1 isextractable (or isolatable or resolvable or segmentable) from theprocessed 3D data to generate a wireframe rendering of all or part ofthe concrete girder box bridge. The present invention provides apreviously unavailable methodology to automatically generate a wireframerendering of complex structures and/or to generate an accurate wireframerendering, such as this concrete girder box bridge, wherein thestructure is considered “complex” at least because it comprises one or aplurality of intersecting quadric surface forms.

By way of distinction from the prior art in one aspect, the inventivewireframe generation process can be considered to be in the first ordera “data-driven” methodology as opposed to a “model-driven” method, assuch terms are discussed elsewhere herein. While 3D data in the form ofpoint clouds is used along with “geometric primitives” to generate the3D information of the structure of interest, the inventive methodologyis not a “hybrid method” as such term in known in the art. In thisregard, hybrid approaches use information derived from the 3D data viasegmentation algorithms etc. to identify features such as boundaries(e.g., ridges and edges), for example, and then apply “geometricprimitives” in the form of expected building elements from a pre-definedlibrary of such expected building elements. Notably, the “geometricprimitives” as meant in the prior art are defined by way of being actualcomponents in building structures or parts of building structures, muchlike “puzzle pieces” that are selected to best fit into the “puzzleoutlines” presented in a generated point cloud incorporating a structureof interest therein.

In contrast, the inventive method extracts (or isolates or resolves orsegments) one or more of a finite list of mathematically generatablegeometric primitives directly from processed 3D data, where suchprocessed 3D data is generated from the methodologies herein. Thepresent method does not find a geometric primitive that fits the pointcloud best; rather, the present method extracts (or isolates or resolvesor segments) a geometric primitive that possesses the same shape of thatset out by the boundary information present in the processed 3D data,which is, by definition, the not the “best fit” but, in fact, the actualfit.

The inventive methodology lends itself to improvements in automaticgeneration of wireframe renderings from the processed 3D data havingboundary data. For example, in some aspects, the present inventionallows accurate wireframe renderings of one or more structures ofinterest in a scene to be automatically generated substantially withoutthe need to first reference a separate library of shape primitives,solid building primitives, pre-defined object shapes, or description ofa structure using a formal grammar against point clouds and/or theimages including the structure of interest. The ability to directlyidentify or isolate or resolve or segment geometric primitives from the3D data having boundary information therein, that is, 3D and/or 2D dataprocessed according to the methodology herein, to generate wireframerenderings enables structures that have not previously been incorporatedinto building models or object libraries to be processed into wireframethat are true to the original form of the structure of interest. Suchgenerated wireframes can be one or more of substantially accuraterepresentations of the surfaces and surface boundaries of the structureof interest, and numerically accurate, topologically accurate, andsemantically accurate. In other words, the present invention enablesaccurate wireframe renderings of sui generis or “arbitrary” structuresof interest to be generated, whether automatically or otherwise.

In prior art methods in which geometric primitives are utilized toderive building models from structures, confirmation that the derivedwireframe renderings, such as various features or shapes present on aroof, for example, is conducted using look ups in a shape library thatis accessed during processing into a wireframe. This necessarilyrequires the structure of interest or elements comprising that structureof interest to be present in the library for the resulting informationto be accurate. Notably, the present invention provides significantimprovements in the ability to generate accurate wireframe renderingsfrom structures that may be substantially non-symmetrical and/orcomprised of unique or non-conforming shapes. Using prior art methods,such shapes can present challenges because it is unlikely that a libraryof shapes used to generate the wireframes will include the subjectshape. The present invention allows suchasymmetrical/unique/non-conforming shapes to be successfully generatedinto wireframe renderings. The present invention therefore allowsstructures of interest to be accurately processed into wireframerenderings even when a corresponding confirming shape is not present ina look up library. This aspect enhances the ability to obtain structureinformation substantially automatically because a reference shape is notnecessary to ensure that the resulting wireframe is correct. For atleast this reason, the present invention represents substantialimprovement over prior art methodologies.

Moreover, the present invention allows structures that include custom orunexpected features to be accurately represented as wireframerenderings. For example, a structure of interest may incorporate acombination of atypical features, where it is unlikely that apre-defined library of objects may include these features incombination. Any attempt to generate an accurate wireframe rendering ofthis unique combination of features using prior methodology will requireat least some human supervision. This can be contrasted with theinventive methodology wherein such atypical features can, in someaspects, be resolved substantially without human supervision to generatean accurate wireframe rendering of the structure of interest.

Moreover, prior art methods often ignore the recognition and modeling ofstructures such as dormers, chimneys, doors, windows, and other buildingparts like balconies, oriels, and stairs due to the inherent complexitythat these structures impose to the problem. The complexity of thesestructures arises from the fact that they are typically enclosed by alarger structure or element of interest. The present invention accountsfor hierarchical relationships that might exist between individualsurfaces. Accordingly, in some aspects, at least two geometricprimitives will be extractable (or isolatable or resolvable orsegmentable) from the processed 3D data, where at least part of theboundaries of one of the geometric primitives is incorporated within theboundaries of the second geometric primitive.

Another shortcoming of the prior art methodologies is the assumptionthat point cloud data having a minimum noise level is provided as theinput data. Additionally, it is often assumed that the structure ofinterest is already segmented and isolated from the rest of the scene inthe point cloud data. Since point cloud data generated from real-lifeenvironments often contains extra structures and artifacts due to treeclutter, reflection from windows, water features, transparent surfaces,feature-less or poorly-textured surfaces, etc. these assumptions proveto be unrealistic in practical settings. Errors underlying theassumptions inherent in prior art methodologies will often result in anincorrect geometric primitive being selected for fitting to the pointcloud data from the pre-defined library of shapes, thus leading toerrors being propagated in the resulting wireframe renderings.Accordingly, the wireframe renderings generated from prior artmethodologies that incorporate such assumptions will require correctionby manual interventions.

In some cases, a structure of interest can be represented as acollection of geometric primitives from the list of Table 1, whereineach of the geometric primitives is, independently, identifiable,isolatable, resolvable, or segmentable from the processed 3D data. Inthis regard, the geometric primitives together can be represented as asingle geometric primitive having all or part thereof within the surfaceboundaries of another geometric primitive. Still further, a structure ofinterest can be characterized as a wireframe rendering generated from atleast two geometric primitives from Table 1 wherein at least part of afirst geometric primitive is within at least some of the surfaceboundaries of a second geometric primitive. The geometric primitives canbe the same or different. An example of different geometric primitivescan comprise a first plane within a second plane is the case of askylight present on a roof. In this case, there will be two planes withouter sections, and additional or iterative processing of the processed3D data could be needed to generate the wireframe rendering and/orresolve the measurements of each of these planar sections. An ellipsoidor other type of rounded geometric primitive could be fully or partiallyincluded in the surface boundaries of a plane. An “eyebrowed” roofdesign, for example, is an example of a situation where a roundedgeometric primitive can be incorporated in a geometric primitive that isa planar. The inventive methodology can allow accurate wireframes foreach of the structures of interest to be generated therefrom.

FIG. 1 illustrates an implementation of the inventive methodology togenerate wireframe renderings of a structure of interest in a scene. Inan exemplary wireframe generation process 100, in 105 the presentmethodology receives 2D data and 3D point cloud data as input for thewireframe generation process, where such data is referred to as“processed 3D data having boundary information” elsewhere herein. Asdiscussed previously, such processed 3D data can be generated from 3Dpoint clouds and 2D image data associated with such point clouds. Theprocessed 3D data can also comprise 3D point clouds without associatedimage data, where the 2D data incorporated therein is generated from thevirtual/synthetic view process to provide the processed 3D data havingboundary information therein.

As an overview, in 110 an optimum resolution is calculated, in 115 thatresult is transferred to a hyperspace, in 120 a surface continuity imageis generated, followed by the finding of connected regions in 125,whereby the structure of interest is isolated in 130. Once the structureof interest is isolated from the processed 3D data, 3D data associatedwith 2D is processed in 200-235, specifically in the steps of:calculating optimum resolution in 205; transferring the data from 205 toa hyperspace; generating a surface continuity image in 215; finding allgeometric surface patches in 220; merging geometric surface patches togenerate geometric surfaces in 225; finding and optimizing concave hullsfor each geometric surface in 230; and combining and optimizing thecollection of concave hulls (i.e., surface boundaries). A wireframerendering of the structure of interest is generated in 300.

Certain segmentation and related steps are conducted in both 105 (whichoperates to isolate the structure of interest from the processed 3Ddata) and 200 (which operates on the isolated structure of interest) inorder to reduce noise and increase accuracy. A prerequisite for suchsegmentation is to generate a surface continuity image in which can thenbe segmented into regions of interest according to elevation, surfacecontinuity, overlap and occlusion, surface area, enclosing concave hullshape, geo-location data, and/or the like. These regions can then betransferred into the coordinate system of the given 3D data and the datacould be segmented accordingly.

Since the 3D data can be provided in absolute scale or it can beprovided in an unknown scale (as can be the case in point cloudsgenerated using image-based solutions without utilizing geo-locationdata or ground control points), a scale-invariant process can beapplied. For example, a wireframe generation process applied to a pointcloud having in an arbitrary scale can be as successful as the wireframegeneration process for the 3D data provided in the absolute scale. Forthis purpose, an optimum resolution can be calculated for the entire 3Ddataset and then the 3D data is transferred to a hyperspace that isinsensitive to the scale of the 3D data.

Once the 3D data is transferred into the hyperspace at 115, a surfacecontinuity image can be generated at 120 that highlights anydiscontinuity in surface coordinates or parameters, normal vectors,overlapping surfaces, occlusion, and density of the data in the givenresolution. A local maxima in the surface continuity image can be a seedpoint to isolate an structure of interest at 130. Having a seed point,seed-based segmentation algorithms could be used to find a connectedregion in 125 in the surface continuity image and then transfer that toan enclosed volume in the 3D data. In this regard, let I define thesurface continuity image and p(u, v) with (u,v)∈R² be a scaler functionthat defines the image. The morphological gradient of I is δ_(D) _(p)=(p·D)−(pºD) where (p·D) and (pºD) are the elementary dilation anderosion of p by the structuring element D. The Laplacian in this case isgiven by Δ_(D) _(P) =(p·D)−2p+ºD). Each connected region is a structureof interest and is processed separately in the following steps in orderto extract the wireframe.

For each segmented structure of interest in the 3D data (wheresegmentation thereof occurs in steps 205-215), local geometric surfacepatches can be extracted in 220 according to canonical expressions thatwere presented for twelve scenarios in Table 1. This exploits the localgeometric relationships among points on the same quadric surface. Thissegmentation clusters the 3D data using lossy-compression. Assuming

is from a mixture of Gaussians and putting a segmentation into Kclusters as

={{acute over (w)}₁ ∪{acute over (w)}₂∪ . . . ∪{acute over (w)}_(K)},then the total number of bits to encode

up to distortion λ is

L S = { 1 , 2 , ⁢ . . . ⁢ , ⁢ K } = ∑ i = 1 K ⁢ [ L ⁡ ( i ) - | i | log 2 ⁡ (| i | ) ]

The 3D data in each segmented group should be then transformed to acanonical pose to improve the conditioning of the quadric modelparameter estimation. For this purpose, the 3D data is first translatedby subtracting the centroid, followed by a Singular Value Decomposition(SVD). The rigid transformation is then calculated from the centroid anda unitary matrix from the SVD which can be viewed as a matrix in SO(3).Afterwards, a fully quadric model is fit using a least-squareformulation. This model is

⊖

==0; ⊖∈

^(10×10), which can be expanded as

ax ² +by ² +cz ² +fyz+gzx+hxy+px+qy+rz+d=0.

The least-square optimization of this model can be performed usingMLESAC, which is an accurate and robust method in the presence ofmeasurement uncertainty and noise.

The local geometric surface patches are then merged in 225 and expandedbased on a model merging process to create 3D surfaces. This is aniterative process and merges the points from the same surface primitivesfollowed by re-estimating the quadric model. Concurrently, thedecision-making process can be cross-referenced with the surfacecontinuity image so that the overall integrity of the data is preserved.Noise level in the 3D data can comprise a significant factor inidentifying the appropriate primitive geometry parameters to a localsurface patch. As can be seen in practice, the noise level in 3D datafrom expensive equipment like laser scanners is negligible compared tothe noise level from inexpensive capture methods such as photogrammetryand image-based 3D reconstruction. This is ignored in the prior artmethods as they assume 3D input data with a certain noise level, thusreducing the quality of wireframe generation from photogrammetry andimage-based 3D reconstruction. The present invention substantiallyeliminates this concern by proposing an adaptive threshold approach fordefining the acceptable lower and upper threshold levels in fittingmodel parameters to the set of observed data. Each geometric surfacepatch therefore has its own threshold values and the conflicts amongintersecting surface patches are resolved through maximizing alikelihood function. This makes the entire wireframe generation processresilient and robust notwithstanding the noise level in the input 3Ddata. Accordingly, the present invention can allow accurate 3D wireframerenderings to be generated from 3D data generated from devices otherthan laser scanners, such as passive image capture devices.

Surface continuity images can help to eliminate noise present in the 3Ddata that, if present, could reduce the accuracy of the resultingwireframe. A surface continuity image can be generated to allowfiltering of the primitive geometric surfaces such that the noise inboundaries can be minimized and to ensure that there is substantially noredundant/wrong segments attached to the primitive geometric surface(s)from which the wireframes are rendered.

A concave hull or alpha shape can be assigned to each extracted 3Dsurface in 230. The concave hull defines the 3D surface boundary and isthe primary component for generating a wireframe rendering for astructure of interest. Rule-based reasoning appropriate for a specificapplication can also be used to assign a polygon to each boundaryfollowed by simplifying the resulted polygon, where the applied rulesare derived from specific conventions that are relevant. For example,when the structure of interest is a building, the applied rules canprimarily be adapted from standard construction practices andconventions. In this regard, specific application in which the inventionherein will be used, will have normal or typical characteristics, suchas roof angle, length, pitch, and shape conventions or the like. Aseries of shape optimization or shape inference rules could further beapplied to minimize noise, enhance consistency of polygons, and enforcesimilarity/parallelism/symmetry. As an exemplary process, this couldinclude projecting the concave hull into 2D images and then evaluatingthe validity of the concave hull shape based on pixel region propertiessuch as intensity change, spatial derivative, texture, and the like. Thesame properties could be further used to fine-tune the projected 2Dconcave hull and then transfer the optimum shape from 2D to the 3Dspace.

If appropriate, a hierarchical relationship can be established among thedetected surfaces to ensure the correct representation ofsuperstructures and parent-child features such as roof dormers,chimneys, windows and doors, stairs, and the like. As an exemplaryprocess, detected surfaces can be projected into a base surface. If thebase surface wholly or partially encloses a projected surface and theprojected surface satisfies some predefined geometrical constraints(e.g., shared vertices, shared edges, small distance to the basesurface, parallelity of the surfaces, perpendicularity of the surfaces,etc.), a parent-child relationship is established. These relationshipscould have multiple depths among different surfaces and hence create ahierarchical relationship.

In 235 individual surface boundaries (i.e., polygons) are then combinedto create the wireframe of the structure of interest in 300. In thisstep, shared polygon vertices, intersecting edges, overlapping edges,intersecting planes, overlapping planes, intersecting surfaces,overlapping surfaces, etc. are detected. Several other criteria can beused in merging the adjacent edges on different polygons such as spatialproximity, direction of the edges, length of the edges, geometry of theconnected polygons, symmetry of the structure, etc. A 2D and/or 3Dglobal optimization (e.g., multi objective linear optimization) can bedefined accordingly by combining all the previously mentioned rules thatwere used to optimize individual surfaces/polygons. This globaloptimization can ensure the integrity of the wireframe for the wholestructure of interest. The outcome is a wireframe that presents aninformation-rich abstraction of a structure of interest.

Although not shown on FIG. 1, once generated, individually optimizedconcave hulls or polygons generated may optionally be further be sentinto a machine learning pipeline that is trained over fine-tuning the 3Dcoordinates of polygon vertices and/or their appearance in 2D images.One exemplary process can be projecting each 3D vertex onto images orvirtual views that include the given vertex. The projected coordinatescan then be fine-tuned such that the cornerness probability ismaximized. The fine-tuned 2D coordinates are converted into 3Dcoordinates by applying the visual triangulation technique discussedherein above in relation to generation of “virtual/synthetic views.”Such information generated from machine learning processes can be usedin training sets etc., as discuss hereinafter.

In a separate inventive aspect, the processes herein allow accuratewireframe renderings of one or more structures of interest in a scene tobe generated directly from a plurality of 2D images without thegeneration of point clouds as an interim step. In other words, awireframe rendering is derivable directly from 2D image data includingthe structure of interest. Such methodologies can be useful to, forexample, generate wireframe renderings that require less computationresources than are required for wireframe generation via point clouds.

In this aspect, machine learning models and computer vision techniquesare utilized to generate 3D data from which wireframe renderings can begenerated using substantially only images and/or camera parametersderived from the 2D images. In this regard, machine learning models andcomputer vision techniques used can include, but is not limited to,validation of the information present in a scene, accuracy ofinformation about a scene, presence of objects of interest in scene,location of objects of interest in a scene in relation to each other,location of objects in a scene in 3D space, labels of objects ofinterest, counts of objects of interest, and/or grouping of objects ofinterest. These techniques can include as non-limiting examples: neuralnetworks, regression models, and/or binary classification models.

This aspect of present invention further includes the continuous orperiodic training of machine learning models based on informationautomatically generated by operation of the present invention oradditional information that may be, but does not need to be provided byusers, outside data sources, or already trained machine learning models.

FIG. 2 illustrates an implementation of the inventive methodology. In anexemplary process herein using a combination of machine learning andcomputer vision techniques combined with use of camera parameters, andimages or videos incorporating the structure or object of interest, thepresent methodology can allow the generation of wireframe rendering foreach structure of interest, counts of objects, locations of objects inrelation to one another or in the real world when GPS data is availablewith no need for pre-processing of the 2D image data to generate pointcloud data as an interim step. The process is separated into logicalsteps of 2D information processing, extraction of points of interest,determination of relation of points of interest to create structures ofinterest, and turning structures of interest into wireframe renderings.

As illustrated in FIG. 2, the process 200 is as follows. 2D images areprovided in 205 for image processing in 210. An implementation of suchimage processing 210 is reading, by the computer, image data includingmetadata in 215. Such metadata can include, for example, GPS data,intrinsic and/or extrinsic camera parameters, image height, and/or imagewidth.

From image processing step 210, points of interest derivable from theimages are identified in 220. In this regard, in 225 a plurality ofpredictions is generated and aggregated via one or more machine learningmodels comprising one or more training sets pertinent for making aspectsof the 2D images that might comprise potential structures of in thescene. For example, machine learning algorithms having training setsrelevant for the identification of building elements (e.g., roofs, roofparts, doors, windows, etc.) or building interior elements (e.g., doors,windows, floors, walls, mechanical systems, furniture etc.) can be usedto generate predictions in 225, which are then, optionally, analyzedagainst rule sets and determining confidence levels in 230. Theparameters of 230 can be a function of how many false positives, falsenegatives, or how far off by some measure of accuracy a prediction canbe while being considered acceptable by later steps or users of theprocess. Specifically, a rule set may or may not be applied in 230depending on the baseline parameters ascribed to determine the accuracyof the prediction. The outputs of the machine learning models will thenbe aggregated across all objects and all images to be fed into one ormore computer vision algorithms to identify points of interest in 235,whereinafter the point of interest is determined by object type in the3D context in 240. A variety of algorithms can be applied in 235 toprovide additional information such as the triangulation of objects in3D based on their predicted position is 2D space from multipleviewpoints in a scene. The point of interest by object type is theoutput of 240.

It should be noted that the output of one aspect or all of step 220 canbe useful on its own and may be provided as output in the case ofinterest in strictly 2D information, where such generated predictionscan be used in machine learning methods, such as the determination of 3Dlocation of objects amongst one or more other objects. Such generatedinformation may then be fed into additional computer vision algorithms,machine learning algorithms, or rule sets in order to combine theinformation into a new set of information which provides context to howthe detected objects interact with one another to thereby generate oneor more meaningful, higher-level structures. From these generatedstructures, the information can be turned into wireframes or any othertype of meaningful information based on 3D positioning, labels, orcontext of the structure. Furthermore, in some aspects, the presentmethodology is scale invariant. This means that the surface boundary orwireframe generation process for 3D data having an arbitrary scale canbe as successful as the process on 3D data with absolute scale.

In 245, the structures of interest are identified by determiningmeaningful interactions or proximities of the data in 250, followed byprocessing in 255 to determine confidence levels whereby data is kept ordiscarded. Wireframe renderings for the structure(s) of interest are theoutput 260 of process 200.

Similar machine learning workflow or logic could be used to auto-draft astructural sketch from one or a plurality of 2D images. As anon-limiting exemplary process, a single aerial orthographic image froma roof structure could be fed into the system. The output could be a 2Dorthographic wireframe that includes 2D coordinates of corner points,edges, connected surfaces, edge dimensions, measurements, edge types,and the like. Measurement information in this orthographic wireframe isaccurate for edges that are perpendicular to the orthographic projectionvector. Other measurement information will have varying levels of errordepending on the angle of the edge with the orthographic projectionvector which is an unknown value; higher angle difference would lead tohigher errors. One exemplary method to completely eliminate such anangle-dependent error is to manually provide the algorithm with theangle value for each edge allowing the algorithm to calculate the offsetand accordingly precise measurements.

Regardless of how the wireframe renderings are generated (e.g., via 2Dand/or 3D data), the generated wireframe renderings and any dataassociated therewith can also optionally be sent into a machine learningpipeline that has been trained on fine-tuning the 2D and/or 3D geometryof a wireframe and its appearance in 2D images. As would be recognized,such machine learning pipelines will be conducted fully automatically bythe computer. Such relevant geometry could include vertices, edges,global or local 3D coordinates, length, direction, angle, surface area,enclosed volume, geo-coordinates, connectivity of different surfaces ofone or more structures, occlusion and overlap between surfaces, and thelike. The machine learning pipeline can be configured to be able toidentify inconsistencies in the overall geometry of the wireframe and tofurther fine-tune the geometry to ensure the wireframe is geometricallyand dimensionally accurate prior to use thereof in engineering-relatedapplications where accuracy is an important aspect. In this regard, whenthe information derived from the wireframe renderings, is placed into amachine learning process incorporating relevant training sets,improvements can be generated in the algorithms used to process the 3Ddata which will, in turn, result in improvements to the subsequentgeneration of wireframe renderings.

In further aspects, the methods can incorporate the step of checking thecharacteristics of the resulting polygon(s), or portion thereof,obtained for each extracted (or isolated or resolved or segmented)primitive geometric surface against a library of existing shapes whileor after the wireframe is generated. Such confirmation can beincorporated into stored information associated for use. Note that sucha library of existing shapes is referenced after one or more geometricprimitives are first isolated or resolved from the 3D data. Such libraryof existing shapes can be provided as a result of information generatedfrom machine learning processes, as discussed elsewhere herein.

Furthermore, the geometry of identified surfaces in wireframe renderingsmay be embedded in an object-based representation format. In thisformat, each surface can be represented with an object that encompassesthe geometrical properties of the surface using several features orvariables. Topological relationships among the identified surfaces couldbe further added to the object-based representation. This could includeconnectivity, occlusion, overlap, logical and/or binary relationships(e.g., a skylight is on top of a roof plane or cutout relationshipbetween a window and a wall), constraints in a practical setting (e.g.,a wall is perpendicular to a ceiling), and the like. This can be furtherexpanded to include the semantic information for each surface includingbut not limited to object label, edge type, material type, materiallayers constructing the surface, unit cost of material or labor, maker,model, date, heat transfer coefficient, strength of the material, fieldnotes, maintenance data, maintenance history, etc. Such a comprehensiverepresentation can transform a CAD model into the more detailed BIMmodel. The present invention therefore comprises generating a BIM modelfrom a CAD model using the methodology herein. Once again, machinelearning models can optionally be used to better ensure the consistencyand validity of the resulting wireframe renderings for use in BIMmodels, as well as fine-tuning of examples that deviate from theiroptimum state.

When more than one structure of interest is present in a scene, theinventive processes can be repeated for each structure of interest sothat a wireframe rendering can be generated for each of the separatestructures of interest. Such individual wireframe renderings can then bemerged/combined to create an overall wireframe which is a representationas a collection of all structures of interest in the scene. A globaloptimization problem can then be formulated to optimize the entirescene, and the structures of interest incorporated therein based on oneor more of a spatial proximity, direction of the edges, length of theedges, geometry of the connected polygons, symmetry of the structure,etc.

By way of example, given a number of 2D images and/or 3D point cloudsgenerated from a scene that can be processed according to themethodology herein, the appearance of the scene from a new viewpointthat did not exist before can be predicted. These transformations can beconfigured to directly operate on images and recover scene informationthat is appropriate to accomplish the desired effect. Informationgenerated therefrom can be incorporated into computer vision algorithms,such as those used in the process described in FIG. 2 herein, as well asin other implementations.

Still further, such generated transformations could also be useddirectly to process 3D points derived from a point cloud to therebygenerate 2D representations through perspective geometry methods, suchas by generating virtual/synthetic views from point clouds where no 2Dimages are associated with such point clouds. Such generated 2D can thenbe used with the parent point clouds to provide 3D data having boundaryinformation therein. Such would be the case from a point cloud generatedfrom LiDAR where no images are associatable therewith. The number ofvirtual/synthetic views that could be generated is virtually limitlessas there is no restriction on intrinsic and extrinsic parameters ofthese views.

In the case of creating virtual/synthetic views from a 3D point cloud, aset of intrinsic and extrinsic parameters can be selected for a desiredvirtual/synthetic view with respect to the coordinate system of thegiven point cloud. Once these parameters are fixed, a 3D transformationcould be calculated that converts 3D coordinates of each point in thepoint cloud into 2D coordinates in the image plane. The colorinformation of the point from the 3D point cloud can then be assigned tothe projected 2D point. A z-buffer approach can also be utilized toaccount for visibility, occlusion, and different depth values for pointsthat fall into the same pixel coordinates. This allows assigning a depthvalue to each pixel in the virtual/synthetic view. Depending on theexistence of color or gray-scale point clouds and utilization of thedepth value for each pixel, the generated virtual/synthetic view/imagecould be one or a plurality of the following types: gray-scale depthimage, gray-scale image, color depth image, color image, multi-layergray-scale depth image, multi-layer gray-scale image, multi-layer colordepth image, multi-layer color image, etc. In the case of unavailabledata for certain pixels in the virtual/synthetic view/image, the emptypixels might be filled using an inpainting image repair approach. Thenotable advantage of such complete virtual/synthetic view/image is thatthey can be directly used in any algorithm that consumes different imagetypes as input data (such as feature detection, object identificationand labeling, object segmentation, region growing, etc.) as if there wasa real camera capturing the scene from that viewpoint.

Illustration of the methods in practice can provide context for theinvention. In relation to roofing applications, which is presented as anon-limiting example, wireframe renderings derived from 3D data having2D information associated therewith, such as a point cloud and aplurality of 2D images of including a roof of interest, can begenerated. In this regard, a 3D point cloud having boundary informationtherein that is derived from processing images and scans of an actualroof will comprise information therein related to each of themeasurements or dimensions of the roof (e.g., lengths on the back, frontand sides of the structure, any features such as gables or hips or eavesor ridges, bounding relationships between roof parts, and thepitches/angles or azimuth thereof etc.). Individual structures orcollection of structural components present in the processed 3D data canbe extracted (isolated, resolved, or segmented) as geometric primitivesas discussed previously, where such geometric primitives are selectedfrom the group consisting of those in Table 1. Such geometric primitiveinformation can be used to define surface boundaries, so as to allow oneor a plurality of areas on the roof to be generated as one or morewireframe renderings using the hereinabove described processing steps.Such wireframe renderings, which can be accurate as to measurements,dimensions, geometry etc., can be generated directly from 3D datagenerated substantially without manual intervention by a user.

In roofing applications, for example, the methodology herein could beaugmented by incorporating methods for matching roof dimensions asmeasured during different times of roofing construction process or bydifferent sources of measurements. For example, prior to ridgecapplacement, the dimensions of relevance are evaluated in relation to theroof plane, but once the ridgecap is in place, roof dimensions are takenfrom top of ridgecap to corner. Yet further, accurate information aboutplanar components in at least one structure of interest can be derivedfrom scanning or imaging substantially without attendant knowledge ofthe pitch, angle, etc. associated with the planar aspects present in thestructure(s). In other words, the pitch of a plane as present in theoriginal structure of interest can be obtained directly from themethodology herein. Moreover, accurate information about a collection ofplanar structures and their relationships with and between each other tomake up the total roof structure can be generated directly from a pointcloud generated from scanning or imaging of the roof. Each of theseplanar structures can be processed as set out further herein to renderthe accurate wireframe comprising the collection of planes. More complexstructures, such as a roof having multiple pitches, gables, hips,non-planar features, and the like can also be generated into an accuratewireframe directly from data derived from scanning or imaging that roof.

In relation to generating wireframe renderings of all or part of abuilding façade, wireframe renderings derived from imaging or scanningof the façade, will comprise each of the measurements or dimensions thefaçade and any sub-structures therefrom (lengths on the back, front andsides of the structure, any features such as windows, doors etc.).Individual structures or collections of structures associated with thefaçade can be represented as extracted (or isolated or resolved orsegmented) geometric primitives selected from the list in Table 1, asdiscussed previously. Wireframe renderings, which are accurate, can begenerated directly from therefrom substantially without manualintervention by a user.

In relation to generating wireframe renderings of structures in aninterior setting, such as a lobby, wireframe renderings that are derivedfrom 3D data are derived from imaging or scanning of the lobby willcomprise each of the measurements or dimensions the lobby and anysub-structures therefrom (e.g., lengths or heights of walls, floorsetc., any features such as windows, doors etc.). Individual structuresor collections of structures can be represented as isolated or resolvedgeometric primitives as discussed previously. Wireframe renderings,which can be accurate, can be generated directly from the geometricprimitives substantially without manual intervention by a user.

In relation to generating information about structures in an interiorsetting, such as a floor plan, wireframe renderings generated from 3Ddata that are derived from imaging or scanning of the floor plan willcomprise each of the measurements or dimensions the floor plan and anysub-structures therefrom (e.g., lengths or heights of walls, floorsetc., any features such as windows, doors etc.). Individual structuresor collections of structures can be represented as isolated or resolvedgeometric primitives as discussed previously. Wireframes, which can beaccurate, can be generated directly from point clouds etc. substantiallywithout manual intervention by a user.

While specific examples of roofs, building facades, lobbies, and floorplans have been provided, it should be understood that the type ofstructures, and parts of structures for which wireframe renderings canbe provided using the methodology herein is expansive. As long as one ormore of the specified geometric primitives listed in Table 1 can beidentified, isolated, resolved, or segmented from the processed 3D datahaving boundary information therein, wireframes can be rendered.Moreover, in significant aspects, such wireframe generation issubstantially or totally automatic, and such wireframes are accurate, assuch term is defined elsewhere herein.

As mentioned previously, 3D data from which the wireframes can bederived can comprise point clouds, polygon meshes, or vector models. Thefollowing section provides additional description regarding the sourcesof 3D data prior to incorporation of the 2D data therein to provideprocessable 3D data.

As used herein, a “point cloud” is a set of data points in the samecoordinate system. In a three-dimensional coordinate system, thesepoints are usually defined by X, Y, and Z coordinates. In some aspects,a point cloud will be generated to provide information about theboundaries of the object(s) for which information is incorporated. 3Ddata for use herein can be generated from point clouds incorporating thestructure of interest. Point clouds suitable for use in the presentinvention can be generated by one or more methods known to those ofordinary skill in the art. Additional information about point cloudgeneration is provided hereinafter.

Suitable point clouds can be generated from a plurality of 2D images ofthe scene having at least one structure of interest incorporatedtherein, wherein the plurality of 2D images are generated from a singlepassive image capture device. In this regard, point clouds suitable forprocessing according to the methods of the present invention can begenerated according to the methods disclosed in the '517 patent,previously incorporated by reference, the disclosure of which isincorporated in its entirety by this reference. In particular, the '517patent describes point cloud generation from, in some aspects, a singlepassive video camera where the camera is moving through the scene, andthe processing thereof to generate point clouds having the featuresdescribed therein.

When the 3D data are derived from a plurality of 2D images taken from ascene comprising the one or more structures of interest, a variety ofimage capture device configurations can be used to generate a pluralityof 2D images suitable for use herein, including image capture devicesintegrated into a device such as a smartphone (e.g., iPhone® orGalaxy®), tablet (e.g., iPad® or Amazon Fire®), autonomous capturedevice (e.g., drone or robot), or a wearable device or the image capturedevices can be as stand-alone camera device (e.g., a GoPro®). The atleast one, or one or more, image capture devices can also beincorporated in a specialized measurement device.

Point clouds derived from stereographic image capture methodologies canalso suitably be used. Yet further, other forms of stereographic imagingcan be utilized to generate suitable point clouds for use herein, suchas that disclosed in U.S. Pat. No. 8,897,539, the disclosure of which isincorporated herein in its entirety by this reference.

Point clouds derived from structured light imaging devices e.g., thefirst version of Microsoft Kinect®, Matterport®, Tango®, etc. can alsobe used. As would be understood, such devices combine RGB imaging withdepth detection otherwise known as RGBD images. Such images can beprocessed to generate point clouds using known methods, such asutilizing MATLAB, or open source software libraries, such as the “PointCloud Library.” Yet further, Tango-derived images incorporateinformation derived from motion tracking images with integration ofaccelerometer and gyroscope data to generate detailed data about themovement of the image capture device in space, as well as depthinformation about one or more structures of interest in a scene.Software configured for use with Tango-derived images can be used togenerate point clouds therefrom. Other forms of structured lightinstruments and methods can be used to suitably generate point cloudsfor use herein.

Point clouds generated from time of flight imaging devices are alsosuitable for use herein. As would be recognized, a time of flightimaging device computes the distance or depth value based on the knownspeed of light and based on measuring the time of flight of a lightsignal between the camera and the reflecting object, for each point ofthe resulting image. In a time of flight imaging device, the entirescene is captured with each laser or light pulse. The current version ofMicrosoft Kinect® is a time of flight imaging device.

Yet further, point clouds generated from ground-based or handheld orairborne LiDAR can be used herein. One suitable method for generatingpoint clouds from LI DAR is disclosed in US Patent Publication No.US20090232388, the disclosure of which is incorporated herein in itsentirety.

Point clouds suitable for use herein can also be generated from GPS datacoupled with provided 2D images. For example, when a number of aerialimages having suitable overlap are taken from multiple view anglesgenerated in conjunction with GPS data, a dense point cloud of one ormore objects present in a scene wherein the object(s) are geo-referencedcan be generated.

In some aspects, point clouds from which the wireframe renderings arederived herein are generated from a plurality of 2D images of a scene,where the scene includes all or part of one or more structures ofinterest. At least one passive image capture device can be used togenerate the plurality of 2D images. Yet further, one or more imagecapture devices can be used to generate the plurality of 2D images,where such plurality can include, but is not limited to, multipleseparate capturing devices or camera arrays.

The plurality of 2D images used herein can be obtained from a movingcamera device. Still further, the plurality of 2D images used herein canbe obtained from a video camera. The 2D digital images can be generatedby an image capture device that comprises a passive sensing technique.The image capture devices used to generate the plurality of 2D imagescan be “calibrated” or “uncalibrated,” as such term is defined in the'517 patent, previously incorporated by reference.

As used herein, “video” means generally that the images are taken, forexample, as single frames in quick succession for playback to providethe illusion of motion to a viewer. In some aspects, video suitable foruse in the present invention comprises at least about 24 frames persecond (“fps”), or at least about 28 fps or at least about 30 fps or anysuitable fps as appropriate in a specific context.

In accordance with some aspects of the invention herein, use of aplurality of 2D images derived from video can improve the ease andquality of user capture of the plurality of 2D images for use herein, soas to allow higher quality point clouds to be generated for use herein.As one example of this improvement, the sequential nature of video hasbeen found by the inventors herein to improve the quality of wireframerenderings. Still further, the inventors herein have found that use ofvideo as the source of the plurality of 2D images can allow tracking ofpoints that are inside (i.e., tracking points within the boundaries ofthe images) or outside of the images of the object of interest (i.e.,continuing to track points that are first “followed” when in the imageframe, and then tracking estimated positions of those points no longerin the images intermediate in time (the points have moved outside theboundaries of the images). When those points are in the field of view oflater image frames, the later-followed points can be substantiallycorrelated to those same features in the earlier image frames), wheresuch point tracking provides improvements in the 2D image informationused for processing herein, such as by creating multiple vantage pointsof full or partial views of the given object. Each vantage point canprovide more information which, in turn, can improve the quality ofmeasurement and prediction. Still further, the inventors herein havefound that use of video as the source of the plurality of 2D images canallow tracking of structures in sequential frames. Tracking ofstructures in sequential frames can provide a basis for prediction fromone frame to the next.

While the present invention is suitable for use with image capturedevices that generate a video from which 2D images can be provided, thepresent invention is not limited to the use of video. That is, theplurality of 2D images can suitably be provided by an image capturedevice that provides 2D still images, such as a “point and shoot”digital camera. These images require the minimum amount of overlapnecessary in order to recreate the scene they comprise. The plurality of2D images herein are suitably overlapping. As used herein, “overlapping”in relation to 2D images means individual images that each,independently, include at least one object of interest, where at leastsome of the images overlap each other as to one or more dimensions ofeach of the one or more structures of interest are concerned. As wouldbe recognized, 2D images derived from video will be overlapping. Toprovide suitably overlapping 2D images incorporating the at least oneobject of interest from sources other than video, the individual imagescan be overlapped, where such overlap is, in reference to the at leastone object of interest, at least about 50% or at least about 60% or atleast about 70% or at least about 80% or at least about 90%. In someembodiments, the amount of overlap in the individual images in theplurality of overlapping 2D images, as well as the total number ofimages, will also depend, in part, on the relevant features of theobject(s). In some aspects, such relevant features include, for example,the amount of randomness in the object shape, the texture of and size ofthe at least one object of interest relative to the image capturedevice, as well as the complexity and other features of the overallscene.

As would be recognized, a plurality of still 2D images taken in sequencecan also be defined as “video” if played back at a speed that allows theperception of motion. Therefore, in some aspects, the plurality ofoverlapping 2D images can be derived from a plurality of digital stillimages and/or from video without affecting the substance of the presentinvention, as long as the plurality of 2D images of the scene includingthe one or more structures of interest can be suitably processed togenerate detailed scene and object information from which themeasurements etc. and predictions can be generated.

In some aspects, the plurality of 2D images includes at least two 2Dimages of the scene, wherein each of the plurality of 2D imagesincorporate at least some of the one or more structures of interest. Inother aspects, the plurality of 2D images includes at least 5, at least10, or at least 15 or at least 20 2D images of the scene, wherein aplurality of the 2D images of the scene incorporate at least some of theone or more structures of interest. As would be recognized, the 2Dimages appropriate for recognizing the one or more structures, orgenerating one or more of counts or predicted labels or generating 3Dinformation which can then provide some, all, or none of geometric,topological, semantic, and/or any 3D information for the one or moreobject of interest in a scene will depend, in part, on factors such asthe size, texture, illumination, degree of randomness in the objectshape, as well as the complexity and other features of the overall sceneand potential occlusions of the object of interest, as well as thedistance of each of the one or more structures of interest from theimage capture device.

As noted, the plurality of 2D images generated for use in the presentinvention can be generated from at least one, or one or more, imagecapture devices comprising passive sensing techniques. Yet further, the2D images can be generated by at least one, or one or more, imagecapture devices that consist essentially of a passive sensing technique.As would be understood by one of ordinary skill in the art,“passive-image capture devices” means that substantially no activesignal source such as a laser or structured light (as opposed to cameraflash or general-illumination devices) or sound or other reflective orresponsive signal is utilized to measure or otherwise sense either orboth of the scene and any of the one or more structures of interest.Additional information may be generated from one or more active devicesused in conjunction with the previously aforementioned passive device ordevices. As would be understood by one of ordinary skill in the art,“active-image capture” devices means that active signal source such as alaser or structured light (as opposed to camera flash orgeneral-illumination devices) or sound or other reflective or responsivesignal is utilized to measure or otherwise sense either or both of thescene and any of the one or more structures of interest.

Yet further, the plurality of 2D images are derived from at least one,or one or more, passive image capture devices, wherein the image capturedevice is moving relative to the scene where the structures in the sceneare moving in a rigid body motion. In other aspects, the 2D images arederived from at least one, or one or more, passive image capturedevices, wherein one of the devices is not stationary relative to thescene or the structures. Yet further, the scene and any included one ormore structures of interest can be moving relative to the at least one,or one or more, passive image capture devices in a rigid body motion.Additional images or other data may be derived from one or more activeimage capturing devices which may be stationary or moving as it may behelpful to the process of object identification and detection, as may beincluded as additional aspects of the process.

The image capture devices can be configured to generate the plurality of2D images of the scene and one or more structures of interest fromground, underwater, underground, cosmic or aerial locations, whereaerial imaging can be conducted by, for example, drones, satellites,balloons, helicopters, unmanned aerial vehicles, manned airplanes or thelike. Ground captures can include any capture taken from an autonomousvehicle, planar or legged robot, or any device with terrestriallocomotive capabilities. Examples of underwater captures include anysubmersive autonomous or manned vehicle that can capture in any body ofwater. Cosmic captures, captures taken from space, can be taken bysatellites, or manned and unmanned vehicles. Underground captures can betaken by various imaging techniques that are suitably used there.

3D data that can be used in the processes herein can also includepolygon meshes. A polygon mesh is a collection of vertices, edges, andfaces that defines the shape of a polyhedral object in 3d computergraphics and solid modeling. The faces usually consist of triangles,quadrilaterals, or other simple complex polygons since suchpresentations simplify rendering, but may also be composed of moregeneral concave polygons, or polygons having holes.

Further 3D data from which wireframe renderings can be derived includesvector data models. A vector data model is representation of a sceneusing points, lines, and polygons.

In further aspects, the methods can further include the step of crossvalidating the generated boundaries and polygon edges by projecting themback into images and verifying the existence of dominant edges in theneighborhood. This could be applicable if the point cloud is generatedusing image-based reconstruction techniques so as to perform aconfirmation step in relation to the original images from which thewireframe was derived.

The generated wireframe renderings can be used to process a combinationof 2D image data and 3D spatial data in order to provide training datafor machine learning, image processing, and other types of algorithms.For example, generated wireframe renderings can provide accuraterepresentations of the locations where the vertices, edges, and planesare on the surface of the structure of interest. This information can beused to determine the locations of the corners of structures in 2D imagedata and 3D spatial data. Corners are defined mathematically by thelocations where edges intersect, or otherwise known as the vertices. Thewireframe renderings can be used to provide both labeled data in 3Dspace that are known to be edges, vertices, or planes and labeled pixelsin 2D images that are the corresponding projected versions of theidentified vertices, edges, or planes which can then be used forsupervised, semi-supervised and unsupervised learning purposes.

Further with respect to the inclusion of machine learning operations toat least some of the generated wireframe renderings, machine learningmodels can be trained on features generated from object structures inorder to learn information about said structures. This data can includebut is not limited to images, drawings, point cloud data, wireframes,amongst others. Features in this case can be defined as a scalar orvector value objects estimated directly from the structures orindirectly by applying an algorithm or function on the structure or datarelated to the structure to generate features. Furthermore, features canbe concatenated with other features (either scalar or vector) in orderto produce larger feature vectors. Each element in the feature vectorrepresents a feature dimension. A feature dimension can be defined byeither an estimated feature (e.g., histogram of gradients, distributionof 3D normals, etc.) or a value directly borrowed from the structure(e.g., positions, dimensions, etc.). Algorithms, like machine learningalgorithms, trained on these feature vectors can learn common structures(e.g., the graph defining their connections), common patterns found inlike structures or scenes and their discrete properties like their typeand shape or continuous properties like their dimensions and angles.Furthermore, knowledge gained from said algorithms can be utilized bymodel-driven methods to refine the estimates of the continuous ordiscrete valued properties.

Machine learning models can also be trained on features gathered fromannotated data about a scene. This data can include but is not limitedto images, data-interchange formats (i.e. JSON, XML, YAML), BIM models(i.e. DWG, DXF, IFC, RVT, NWD), or any other automatically or usergenerated file with information about a scene. Features in this case aredefined as but not limited to scalar, vector, label, proximity, and/orcount objects. Features may be read directly from these files orgenerated indirectly from the application of additional algorithmsand/or combinations of features. Algorithms, like machine learningalgorithms, trained on these feature vectors can learn common structures(e.g., the graph defining their connections), common patterns found inlike structures or scenes and their discrete properties like their typeand shape or continuous properties like their dimensions and angles.Furthermore, knowledge gained from said algorithms can be utilized bymodel-driven methods to refine the estimates of the continuous ordiscrete valued properties.

Machine learning models can be trained on features generated from BIMmodels in order to learn and prediction information about said models.Information such as BIM models and the like can be fed into machinelearning models for additional analysis including but not limited tooutlier analysis, similarity analysis, areas of interest, areas thatrequire inspection, and the like. This seed information may be providedfrom an outside source or may have been created by the present inventionat any point in the process. For example, a BIM model of a room may beprocessed by a machine learning model which is able to make predictionsabout the availability of enough electricity to the room based on thepower supply, availability of outlets, and necessity of objects foundinside the room.

Machine learning algorithms along with computer vision algorithms can beapplied to images and camera parameter information to generate 3Dinformation. This 3D information can be used for, but is not limited to,determining vertices and edges of a structure which can be combined tocreate a wireframe of the object. This output can include 3Dinformation, 2D information, and semantic information about both. Thisoutput can also be used as seed input for other parts of the invention.

Tuning of 3D information generated according to the processes herein canincrease its value in downstream applications, such as for use in objectlibraries. Such tuning can be accomplished through machine learningmethods, model-driven methods and a combination of both. Tuning orrefinement can be considered the act of converging on a more precisevalue of a continuous property like dimensions or angles and/or discreteproperties such as labels, groupings, or existence.

Machine learning algorithms can be applied to the generated wireframerenderings in order to tune them in well-known structures. Variouslayers of logic can be incorporated into these algorithms, where eachlayer can return different information about the structure. By providingdifferent predictions, the algorithm can indicate whether sections ofthe wireframe that do not fit into conventional object shapes, sizes, orangles are to be altered, flagged, or removed based on some criteria forexample the prediction of the algorithm. Furthermore, the algorithm canindicate the need to apply additional processing steps to the generatedwireframe.

Model-driven methods, when applied to generated wireframe renderingsgenerated by identification, isolation, resolving or segmenting ofgeometric primitives selected from the list of Table 1, can returnrobust estimates of continuous property types such as generated frommeasures on dimensions or discrete property types such as label, group,or existence. These methods can require an initial condition, in thecase of the wireframe renderings, these can include an intermittentdefinition of the structure, pose, dimensions, and angles. Theaforementioned values can be provided by the disclosed invention or amachine learning algorithm trained on the output from the disclosedinvention or variant thereof. Once the model-driven algorithm isinitiated, it is activated with each iteration updating its estimate ofthe valued property and optimizing the objective function that drivesit. Definition of the objective function involves determining theproperties to be optimized which can be directly measured from theprovided 2D and/or 3D data or some information derived from these.

The machine learning models can also be tuned through continuouslearning of parameters and hyper-parameters from any number of sourcesincluding but not limited to their own output, user based input, inputfrom outside data sources, parameters from previously trained machinelearning models. This tuning can include but is not limited to updatesto weights used in models, updates to hyper-parameters used in trainingor prediction of models, updates to information fed into training ofmodels, and/or updates to the combination of results of models commonlyknown as an ensemble of models.

Again, tuning can be accomplished through a combination of bothmodel-driven and machine learning methods. By defining the objectivefunction based on extracted or transformed information from the input2D, 3D, or both types of data the benefits of the combination of themachine learning and model-driven method can be obtained. Examples ofsuch benefits can include more robust estimates of the continuous ordiscrete level properties or better defined structures like theestimates of the vertices, edges and planes of which the wireframe iscomprised or labels, groups, and counts of objects appearing in a scene

Machine learning algorithms can be trained on images to learn what partsof structures look like their 2D projected form or in their 3D spatialform in order to more accurately capture the wireframe rendering of thestructure. A machine learning algorithm may be trained on commoninstances of patterns of objects in order to learn how to discriminatethem from other objects. These algorithms can then be applied to imagedata of the original structure from which the wireframe rendering wasgenerated. This serves as another example of the extracted informationfrom the input data that can drive the model-driven method or anyalgorithm tuning the wireframe. The predictions of the algorithms can beused to add vertices, remove vertices, adjust vertices, add edges,remove edges, add labels, or flag a vertex for further observation orprocessing based on some criteria.

Machine learning methods can be trained on structures and theircorresponding 2D and 3D data in order to learn a direct relation betweenthem. Thus, predicted wireframe structures can be estimated directlyfrom the input 2D and/or 3D data. These serve as initial conditions forrefinement or, if the conditions of the objective function is met, canbe taken as direct estimates of the wireframe.

Machine learning methods can return semantic information about thestructure of interest. Vertices, edges and planes in the wireframe arethe graph and geometric properties that define the surface structure andshape. However, additional information can be extracted from the 3Dand/or 2D data about them. For example, information about theircomposition, material, function, or relation to the environment. Machinelearning algorithms can be trained on annotated versions of the dataused to generate the wireframe to learn cues in the appearance orextracted features of the data that would allow them to make predictionsabout the semantic information. For example, with respect to roofs, theycan identify planes with penetrations, the tiles of the roof, roofingcomposition (type or material), gutters, or possibly occluding objectslike trees, facades, windows, doors, and any other objects of interest.

Machine learning models can be applied at any point in the currentinvention including but not limited to the initial seed point, fullscale tuning of all results, fine tuning of a single piece of data inthe result at any point in the process or anywhere and at any scale inbetween.

Machine learning models can be swapped, changed, or modified based on acontent aware pipeline of where the invention is in its process, whatcontent was submitted including what type of data and/or context aboutwhat the data is describing i.e., roof, façade, entire structure,outdoor scene, indoor scene, amongst others), and/or the preferredresults (i.e., expected number of false positives, false negatives, orother potentially modifiable outcomes). This can include, but is notlimited to, different model weights based on what is expected to befound in a scene, different models or different numbers of models beingapplied based on what is expected to be found in the scene or what datais provided, different amounts of outputs based on what is thought to beof interest or what data is provided, and/or different hyper-parametersbased on the scrutiny of results expected. For example, a machinelearning model for a roof structure might be significantly differentfrom models for a building interior scene in terms of model weights,number of models, output types, and the like. Therefore, such a prioriinformation about the scene types and it functions could help inselecting the most optimum machine learning model and hence achievingthe most desired outcome.

Information such as V/R and/or A/R scenes and the like can be input intomachine learning for additional analysis such as object detection,object grouping, areas of interest, and/or areas for inspection, amongstothers. This seed information may be provided from an outside source ormay have been created by the present invention at any point in theprocess. For example, A/R scene data of a construction site may be inputinto a machine learning model to determine areas that have been changedrecently but have not been inspected based on available data which wouldinform the necessary parties that an inspection and may be able todetermine what objects require inspection, what types of inspections maybe necessary, and/or who would be best to do the work.

Determining if an object is obstructing the view and possibly inducingerror in the wireframe estimate can be highly relevant to the results.The disclosed invention generated wireframe renderings having accuracycan be dependent, at least in part, on the 2D and/or 3D data used as theinput data. Error can be incurred if such input data is occluded or ifnoise is present. Machine learning methods can be trained to identifythese instances of occlusion and noise presence and return informativeindicators of their presence. Furthermore, they can be used to trainother algorithms which can return confidence measures for the wireframeor can provide additional information to the tuning algorithms thatwould allow them to compensate for the occlusions and noise.

Still further, probabilistic and semantic based modeling can beincorporated into the evaluation of the generated wireframe renderingsto modify the generated surface boundaries such that its conformity tothe actual element or structure is improved. Yet further, a report couldbe generated based on all or part of the metadata extractable from thewireframe such as surface areas, sematic attributes like ridge, gutter,gable, convex and concave, material estimation, cost estimation, volumeestimation, direction of the surfaces with respect to sun, total timefor sun exposure during the day, volume of the structure, etc.

The methods herein can allow substantially accurate measurements andother dimensional aspects of the features or components of interest tobe derived automatically from the 3D data to generate, for example, oneor more wireframe renderings for use in a wide variety of downstreamapplications where accurate dimensional information, geometricinformation, or topographical information about one or more structuresof interest in a scene may be useful.

As more specific, but non-limiting, examples of uses for the wireframesherein, inventorying, construction, merchandising, insuranceunderwriting and claim adjustment, civil engineering, architecture anddesign, building information management, home remodeling, roofing,flooring, real estate listing, gaming, mixed reality, virtual reality,augmented reality, among other things.

With regard specifically to BIM applications, as would be recognized,BIM is a digital representation of physical and functionalcharacteristics of a facility, building, space, etc., which willnecessarily incorporate information about objects present therein. A BIMis a shared knowledge resource for information about a facility forminga reliable basis for decisions during its life-cycle; defined asexisting from earliest conception to demolition. BIM involvesrepresenting a design as combinations of “objects”—vague and undefined,generic or product-specific, solid shapes or void-space oriented (likethe shape of a cone or more), that carry their geometry, relations andattributes. BIM design tools allow extraction of different views from abuilding model for drawing production and other uses. These differentviews can be made automatically consistent, being based on a singledefinition of each object instance. BIM software also endeavors todefine objects parametrically; that is, the objects are defined asparameters and relations to other objects, so that if a related objectis amended, dependent ones will automatically also change. For theprofessionals involved in a project, BIM enables a virtual informationmodel to be handed from the architect or design team to the maincontractor and subcontractors and then on to the owner/operator; eachprofessional adds discipline-specific data to the single shared model.This seeks to reduce information losses that traditionally occurred whena new team takes ‘ownership’ of the project, and provides more extensiveinformation to owners of complex structures.

When used in the BIM context, the systems and methods of the presentinvention can suitably be used to generate information about theobject(s) present in a facility, where such objects compriseinfrastructure, fixtures, materials, utilities, features, components,and the like. The generated measurements, dimensions, geometries,topography, labeling, and semantic information can be utilized toprovide a deep and relevant collection of information set about abuilding or facility, where such collection can be used in BIMapplications. Information can be generated in accordance with themethodology herein for use of each part of a building structure forwhich BIM can be relevant, for example, CAD design, structural analysis,detailing, HVAC, plumbing, electrical, interior finishes, and the like.

Moreover, the methods herein can be used in lifecycle management of afacility, scene, or site in that the presence, absence, or modificationof previously identified objects etc. can be tracked over time as partof the BIM application. For example, the progression assembly of variousinfrastructure within a facility can be tracked in changes in themeasurement, dimensions or topology of information returned can bemonitored in time. The automated, or semi-automated nature of themethodology herein can reduce the need for in-person monitoring of thefacilities and, as such, BIM applications can be enhanced.

Semantic information can be used in accordance with the methodologyherein provides improvements in BIM applications. For example, if anobject is identified as an HVAC system, for example, further informationabout that HVAC system can be generated when such further information isavailable in one or more libraries of data associated. For example, theoriginal design drawings for that HVAC system can be available, as wellas any service records, warranty information, parts lists, etc.

In the context of M/R (mixed reality), which includes A/R and V/R,augmented reality (AR) can be combined with BIM, as well as otherapplications provides a real-time view of a physical, real-worldenvironment in which the view is augmented with computer-generatedvirtual elements, which may include sound, video, graphics and/orpositioning data. Some mobile computing devices provide augmentedreality applications that allow users to see an augmented view of asurrounding real-world environment through a camera of the mobilecomputing device. One such application overlays the camera view of thesurrounding environment with location-based data, such as local shops,restaurants and movie theaters. Incorporation of the methodology hereinin conjunction with A/R can enhance current applications such as byallowing the information extracted from the scenes to be betterutilized. Creating the digital content for the A/R application is onlypart of the challenge. Positioning the digital overlay in the cameraview is another challenge that can be overcome with this methodology.This application can generate dynamic feature points in any scene torecognize where the A/R objects should be in the view. Today, this maybe done with GPS, registration targets, or other computer visiontechnique. However, the ability to better recognize specific objectsprovides more accurate spatial intelligence to overlay the A/R objectsto improve user experience and interface. This level of accurate spatialintelligence can transform A/R applications from location-basedconsumer-focused overlays to more commercial applications focused onproving visualizations for training and educating engineers, designers,architects, and construction workers.

The systems and methods herein can further be used in virtual realityapplications. As would be recognized “virtual reality” is the term usedto describe a three-dimensional, computer generated environment whichcan be explored and interacted with by a person. That person becomespart of this virtual world or is immersed within this environment andwhilst there, is able to manipulate objects or perform a series ofactions. The information generated herein can be used to improve thequality of virtual reality environments. Today, creating a V/Renvironment is extremely time consuming and takes hours or days ofmanual effort. With the ability to automatically detect, identify, andextract 2D/3D objects the time and effort to create a V/R environment ofthe physical world is drastically reduced. Whether it is a 3D model withextracted objects or the ability to stitch together images to create animmersive digital model, the methodology herein can be used to modify ortransform how content for V/R environments is created/generated. Thesetypes of immersive models can be used for but not limited to videogames,real estate walkthroughs, and training/educational programs forcommercial and industrial applications. Most importantly, thisapplication makes it possible for any consumer or commercial user toautomatically generate an immersive V/R model from any passive or activesensor device.

The systems and methods herein can further be used in gamingapplications. As would be recognized “gaming”, or “video gaming”, is theterm used to describe a game played by electronically manipulatingimages produced by a computer program on a television screen or otherdisplay screen. Types of video games include massively multiplayeronline (MMO), simulations, first person shooter (FPS), action, stealthshooter, educational, and other game types. Today, creating a gamingenvironment is extremely time consuming and takes hours or weeks ormonths of data collection and programming by the game developer. Thereis an interest in providing the user, or gamer, with the ability tobring their own location information, or local scene data into thegaming environment, simulating the game taking place in their room orhome or street. This experience could be considered a fully immersivevideo game experience. In this game experience, the player's scene couldbe combined or integrated with the game developer-created scene, orreplace it entirely, and the experience would seem like it takes placein the gamer's scene, e.g., his or her room. The room or aspects orobjects from the room could be integrated or included in the gamingexperience. With the ability to automatically detect, identify, andextract 2D/3D objects and provide semantic information about the objectsusing a passive camera the inventive technology could enable thisimmersive gaming experience. The game would need a predeterminedinterface definition in which scene information is described, since thegame actions would rely on interacting with aspects of, or objects, inthe scene. This interface can be a specification of data content andformat and electronic method for exchanging the data. It can be assumedthat this interface would include basic scene data such as geometry,volume, structure, and appearance. It would also include descriptiveinformation about relevant objects in the scene, including what theyare, topology and where the objects exist in the scene and relative toeach other, geometry, volume and appearance. Examples of gaming actionsbenefiting from object specific information include doors and openingsin which the game allows passage of the user or other characters orobjects, windows in which the user could see through or could be brokenor opened or other actions of a window, or a cup of liquid on a table inwhich the game could cause it to fall over or enable the user orcharacter to pick it up. Additionally, semantic information provided bythe inventive system can enable the game developer to build morepowerful functionality into interaction with the objects, such as weightand whether an object is movable or force required to move it, thematerial from which the object is made and how it should appear or reactto force applied to it, or material and whether it should bounce orbreak. These are just representative examples but there are endlessbenefits from identifying objects and detailed information about theobjects in the scene. There is great value in a gamer being able tobring their scene into the game environment using just a passive camera.The inventive technology could be built into the passive camera or intothe game console or game controller to enable this functionality.

Yet further, the systems and methods can be used to create inventoriesof objects, such as furniture or components of value that are present ina facility or similar environment. For example, information about highvalue components, such as automobiles can be generated from a scene. Inthis regard, the number and type of automobiles present in a storage lotcan be generated using the methodology herein.

Surveying operations can benefit from the inventive technology. With allthe advances in autonomous vehicle navigation, there is a need forwell-defined maps of both populated and unpopulated areas. There isalready a series of literature as far as detection of objects for thepurpose of obstacle avoidance and safety. However, there remains a needfor recreating the space around a moving vehicle with both geometric,topological and semantic information, for example. In this regard, thepresent invention can allow accurate wireframe renderings of an area orlocation proximate to a vehicle of interest in a scene, thereby allowingthe location of the vehicle to be accurately placed in the scene. Insome implementations, a plurality of wireframe renderings can begenerated, in which topological information or other relevantinformation is incorporated therein for various aspects of the scene,including, but not limited to, the location of the vehicle relative toother structures or objects in the scene.

The inventive methodology can also aid in object avoidance forautonomous driving and drones. Recreating a scene and knowing theobjects that occupy in that scene is a complex process that will greatlyaid in the ability for autonomous vehicles to navigate safely andeffectively.

The inventive methodology can also aid in navigating in an unknownenvironment. Navigating an unknown environment can be a time-consumingand potentially dangerous process. This technology can enable autonomouscraft to explore the environment first and recreate the scene accuratelyand with context in order to provide a clear means of navigation.

Still further, the inventive methodology can help first responders tonavigate in an unknown place, a collapsed structure, or find peopleunable to respond. These situations can often be dangerous and full ofconfusion. By leveraging this technology an autonomous craft can be sentin to navigate any unknown areas, find paths through structures whichmay have collapsed or become damaged, provide semantic information aboutdamage, and detect people or animals which may be in need.

As additional applications, the wireframes and related informationgenerated herein can be used to generate takeoff information andconstruction estimations, as well as building images, CAD drawings, siteplans, architectural drawings, building information models, scaledrawings of a building or structure, landscape plans, interior designplans, inventory management plans, virtual tours, input into virtualreality content creation engines, gaming interfaces, etc.

In conjunction with the methods herein, in some aspects, the softwareassociated with the image capture device and/or the hardware into whichthe image capture device is integrated is configured to provide the userwith interactive feedback with regard to the image-acquisitionparameters and/or the structure selection process. For example, in someaspects, such interactive feedback provides information regarding theobject of interest including whether the tracking is suitable to obtaina plurality of overlapping 2D images necessary to provide suitableimages for use herein. In some aspects, such processing is conducted inthe image capture device itself or the hardware in which the device isintegrated (e.g., smartphone, wearable device etc.). In other aspects,the processing is performed “in the cloud” on a server that is incommunication with the image capture device/hardware. In other aspects,the processing is performed on any device in communication with theimage capture device and/or hardware. In some aspects, such processingis performed on both the device/hardware and an associated server, wheredecision-making regarding the location of various parts of theprocessing may depend on the speed and quality that the user needsresults. Yet further, in some aspects, user feedback is provided in realtime, in near real time or on a delayed basis.

Yet further, in some aspects, the user display of the output hereinthereof is configured to provide user generated inputs to facilitate andenhance generation of the plurality of 2D images, the point clouds, andany derived wireframe renderings or the like. In some aspects, such usergenerated inputs can include, for example, the level of detail, aclose-up of a portion of the object(s) of interest and any associatedimage or generated point cloud, optional colorization, a desirable leveldimension detail, etc.

In a further aspect, the software associated with the image capturedevices and methods herein is configured to provide an accuracy valuefor the generated measurements, dimensions, topology, labels, semanticsetc. By reporting a level of accuracy (where such accuracy is derivableas set out elsewhere herein), a user will obtain knowledge aboutaccuracy of the extracted measurement or other dimensional value, or aprobability that the returned label and/or semantic information isaccurate with respect to the one or more structures of interest.

In some aspects, the software associated with the image capture devicesand/or hardware in which the image capture device is integrated isconfigured to elicit and receive from the user a selection of aregion/area of interest in a captured image(s) of the object of interestfrom which point clouds and generated are derived. For example, in someaspects, when a scene in which one or more structures of interest iscaptured, the software elicits and receives selection of specificobject(s) that are recognized in the scene or for which otherinformation can be provided (e.g., measurements, dimensions, topology,labels, semantics). In this regard, the software can return a query tothe user that asks him to confirm that a recognized object(s) is ofinterest. If the user affirms that the indicated object(s) is ofinterest, further information about the object can be returned. In anexemplary configuration of such an implementation, the scene presentedto the user through a viewfinder or screen on the image capture deviceelicits and receives the selection of an object present in the scenesuch as by touch or other type of method. The object of interest can beprovided for selection by a computer or a user.

In some aspects, the methods of the present invention are suitable foruse, and are performed, “in the cloud” (i.e., the software executes onserver computers connected to the internet and leased on an as-neededbasis). (Note that the word “cloud” as used in the terms “point cloud”described as part of the invention is independent of, and unrelated to,“cloud computing” as such.) As would recognized, cloud computing hasemerged as one optimization of traditional data processingmethodologies. A computing cloud is defined as a set of resources (e.g.,processing, storage, or other resources) available through a networkthat can serve at least some traditional data center functions for anenterprise. A computing cloud often involves a layer of abstraction suchthat the applications and users of the computing cloud may not know thespecific hardware that the applications are running on, where thehardware is located, and so forth. This allows the computing cloudoperator some additional freedom in terms of implementing resources intoand out of service, maintenance, and so on. Computing clouds may includepublic computing clouds, such as Microsoft® Azure, Amazon® Web Services,and others, as well as private computing clouds.

Communication media appropriate for use in or with the inventions of thepresent invention may be exemplified by computer-readable instructions,data structures, program modules, or other data stored on non-transientcomputer-readable media, and may include any information-delivery media.The instructions and data structures stored on the non-transientcomputer-readable media may be transmitted as a modulated data signal tothe computer or server on which the computer-implemented methods of thepresent invention are executed. A “modulated data signal” may be asignal that has one or more of its characteristics set or changed insuch a manner as to encode information in the signal. By way of example,and not limitation, communication media may include wired media such asa wired network or direct-wired connection, and wireless media such asacoustic, radio frequency (RF), microwave, infrared (IR) and otherwireless media. The term “computer-readable media” as used herein mayinclude both local non-transient storage media and remote non-transientstorage media connected to the information processors usingcommunication media such as the internet. Non-transientcomputer-readable media do not include mere signals or modulated carrierwaves, but include the storage media that form the source for suchsignals.

At this time, there is little distinction left between hardware andsoftware implementations of aspects of systems; the use of hardware orsoftware is generally (but not always, in that in certain contexts thechoice between hardware and software can become significant) a designchoice representing cost vs. efficiency tradeoffs. There are variousinformation-processing vehicles by which processes and/or systems and/orother technologies described herein may be implemented, e.g., hardware,software, and/or firmware, and that the preferred vehicle may vary withthe context in which the processes and/or systems and/or othertechnologies are deployed. For example, if an implementer determinesthat speed and accuracy are paramount, the implementer may opt for amainly hardware and/or firmware vehicle; if flexibility is paramount,the implementer may opt for a mainly software implementation; or, yetagain alternatively, the implementer may opt for some combination ofhardware, software, and/or firmware.

The foregoing detailed description has set forth various aspects of thedevices and/or processes for system configuration via the use of blockdiagrams, flowcharts, and/or examples. Insofar as such block diagrams,flowcharts, and/or examples contain one or more functions and/oroperations, it will be understood by those within the art that eachfunction and/or operation within such block diagrams, flowcharts, orexamples can be implemented, individually and/or collectively, by a widerange of hardware, software, firmware, or virtually any combinationthereof. In one embodiment, several portions of the subject matterdescribed herein may be implemented via Application Specific IntegratedCircuits (ASICs), Field Programmable Gate Arrays (FPGAs), digital signalprocessors (DSPs), or other integrated formats. However, those skilledin the art will recognize that some aspects of the aspects disclosedherein, in whole or in part, can be equivalently implemented inintegrated circuits, as one or more computer programs running on one ormore computers, e.g., as one or more programs running on one or morecomputer systems, as one or more programs running on one or moreprocessors, e.g., as one or more programs running on one or moremicroprocessors, as firmware, or as virtually any combination thereof,and that designing the circuitry and/or writing the code for thesoftware and or firmware would be well within the skill of one of skillin the art in light of this disclosure. In addition, those skilled inthe art will appreciate that the mechanisms of the subject matterdescribed herein are capable of being distributed as a program productin a variety of forms, and that an illustrative embodiment of thesubject matter described herein applies regardless of the particulartype of signal bearing medium used to actually carry out thedistribution. Examples of a signal-bearing medium include, but are notlimited to, the following: a recordable type medium such as a floppydisk, a hard disk drive, a CD, a DVD, a digital tape, a computer memory,etc.; and a remote non-transitory storage medium accessed using atransmission type medium such as a digital and/or an analogcommunication medium (e.g., a fiber optic cable, a waveguide, a wiredcommunications link, a wireless communication link, etc.), for example aserver accessed via the internet.

Those skilled in the art will recognize that it is common within the artto describe devices and/or processes in the fashion set forth herein,and thereafter use engineering practices to integrate such describeddevices and/or processes into data-processing systems. That is, at leasta portion of the devices and/or processes described herein can beintegrated into a data processing system via a reasonable amount ofexperimentation. Those having skill in the art will recognize that atypical data processing system generally includes one or more of asystem unit housing, a video display device, a memory such as volatileand non-volatile memory, processors such as microprocessors and digitalsignal processors, computational entities such as operating systems,drivers, graphical user interfaces, and applications programs, one ormore interaction devices, such as a touch pad or screen, and/or controlsystems including feedback loops and control motors, e.g., feedback forsensing position and/or velocity; control motors for moving and/oradjusting components and/or quantities. A typical data processing systemmay be implemented utilizing any suitable commercially availablecomponents, such as those typically found in datacomputing/communication and/or network computing/communication systems.

The herein-described subject matter sometimes illustrates differentcomponents contained within, or connected with, different othercomponents. It is to be understood that such depicted architectures aremerely examples, and that in fact many other architectures can beimplemented which achieve the same functionality. In a conceptual sense,any arrangement of components to achieve the same functionality iseffectively “associated” such that the desired functionality isachieved. Hence, any two components herein combined to achieve aparticular functionality can be seen as “associated with” each othersuch that the desired functionality is achieved, irrespective ofarchitectures or intermedial components. Likewise, any two components soassociated can also be viewed as being “operably connected”, or“operably coupled”, to each other to achieve the desired functionality,and any two components capable of being so associated can also be viewedas being “operably couplable”, to each other to achieve the desiredfunctionality. Specific examples of operably couplable include but arenot limited to physically mateable and/or physically interactingcomponents and/or wirelessly interactable and/or wirelessly interactingcomponents and/or logically interacting and/or logically interactablecomponents.

As described above, the exemplary aspects have been described andillustrated in the drawings and the specification. The exemplary aspectswere chosen and described in order to explain certain principles of theinvention and their practical application, to thereby enable othersskilled in the art to make and utilize various exemplary aspects of thepresent invention, as well as various alternatives and modificationsthereof. As is evident from the foregoing description, certain aspectsof the present invention are not limited by the particular details ofthe examples illustrated herein, and it is therefore contemplated thatother modifications and applications, or equivalents thereof, will occurto those skilled in the art. Many changes, modifications, variations andother uses and applications of the present construction will, however,become apparent to those skilled in the art after considering thespecification and the accompanying drawings. All such changes,modifications, variations and other uses and applications which do notdepart from the spirit and scope of the invention are deemed to becovered by the invention which is limited only by the claims as will bepresented.

What is claimed is: 1) A library of information for use in generatingwireframe renderings of one or more structures in a scene, comprising:a. providing, automatically by a computer, 2D and 3D data for at leastone structure of interest in a scene; b. processing, automatically bythe computer, the 2D and 3D data to generate 3D information comprisingan edge or skeletal representation associated with the structure,wherein the 2D and 3D data comprising the at least one structure ofinterest in the scene is generated by: i. providing a plurality of 2Dimages including the at least one structure of interest and at least onepoint cloud including the at least one structure of interest, whereinthe plurality of 2D images is associated with the at least one pointcloud; ii. processing a plurality of point clouds to extract at leastsome 2D data associated with the at least one structure of interest,thereby providing point cloud data associated with 2D data derived fromthe plurality of point clouds; and iii. extracting, automatically by acomputer, at least one geometric primitive from the 3D informationcomprising the edge or skeletal representation each of the at least onegeometric primitive being an object surface of the at least onestructure of interest; and c. incorporating, automatically by thecomputer, information associated with the at least one geometricprimitive in a machine learning library for use in subsequent processesfor generating wireframe renderings of one or more structures fromimages of scenes including each of the one or more structures.